{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DeepHit for Single Event\n",
    "\n",
    "In this notebook we show an example of how we can fit a [DeepHit](http://medianetlab.ee.ucla.edu/papers/AAAI_2018_DeepHit) model when we only have *one* event type.\n",
    "\n",
    "If you are interested in **competing risks**, see [this notebook](https://nbviewer.jupyter.org/github/havakv/pycox/blob/master/examples/deephit_competing_risks.ipynb) instead.\n",
    "\n",
    "For a more verbose introduction to `pycox` see [this notebook](https://nbviewer.jupyter.org/github/havakv/pycox/blob/master/examples/01_introduction.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# For preprocessing\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn_pandas import DataFrameMapper \n",
    "\n",
    "import torch # For building the networks \n",
    "import torchtuples as tt # Some useful functions\n",
    "\n",
    "from pycox.datasets import metabric\n",
    "from pycox.models import DeepHitSingle\n",
    "from pycox.evaluation import EvalSurv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Uncomment to install `sklearn-pandas`\n",
    "# ! pip install sklearn-pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1234)\n",
    "_ = torch.manual_seed(123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "We load the METABRIC data set as a pandas DataFrame and split the data in in train, test and validation.\n",
    "\n",
    "The `duration` column gives the observed times and the `event` column contains indicators of whether the observation is an event (1) or a censored observation (0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = metabric.read_df()\n",
    "df_test = df_train.sample(frac=0.2)\n",
    "df_train = df_train.drop(df_test.index)\n",
    "df_val = df_train.sample(frac=0.2)\n",
    "df_train = df_train.drop(df_val.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x0</th>\n",
       "      <th>x1</th>\n",
       "      <th>x2</th>\n",
       "      <th>x3</th>\n",
       "      <th>x4</th>\n",
       "      <th>x5</th>\n",
       "      <th>x6</th>\n",
       "      <th>x7</th>\n",
       "      <th>x8</th>\n",
       "      <th>duration</th>\n",
       "      <th>event</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>5.603834</td>\n",
       "      <td>7.811392</td>\n",
       "      <td>10.797988</td>\n",
       "      <td>5.967607</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>56.840000</td>\n",
       "      <td>99.333336</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>5.284882</td>\n",
       "      <td>9.581043</td>\n",
       "      <td>10.204620</td>\n",
       "      <td>5.664970</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>85.940002</td>\n",
       "      <td>95.733330</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>6.654017</td>\n",
       "      <td>5.341846</td>\n",
       "      <td>8.646379</td>\n",
       "      <td>5.655888</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>66.910004</td>\n",
       "      <td>239.300003</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5.456747</td>\n",
       "      <td>5.339741</td>\n",
       "      <td>10.555724</td>\n",
       "      <td>6.008429</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>67.849998</td>\n",
       "      <td>56.933334</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>5.425826</td>\n",
       "      <td>6.331182</td>\n",
       "      <td>10.455145</td>\n",
       "      <td>5.749053</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>70.519997</td>\n",
       "      <td>123.533333</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         x0        x1         x2        x3   x4   x5   x6   x7         x8  \\\n",
       "0  5.603834  7.811392  10.797988  5.967607  1.0  1.0  0.0  1.0  56.840000   \n",
       "1  5.284882  9.581043  10.204620  5.664970  1.0  0.0  0.0  1.0  85.940002   \n",
       "3  6.654017  5.341846   8.646379  5.655888  0.0  0.0  0.0  0.0  66.910004   \n",
       "4  5.456747  5.339741  10.555724  6.008429  1.0  0.0  0.0  1.0  67.849998   \n",
       "5  5.425826  6.331182  10.455145  5.749053  1.0  1.0  0.0  1.0  70.519997   \n",
       "\n",
       "     duration  event  \n",
       "0   99.333336      0  \n",
       "1   95.733330      1  \n",
       "3  239.300003      0  \n",
       "4   56.933334      1  \n",
       "5  123.533333      0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature transforms\n",
    "\n",
    "The METABRIC dataset has  9 covariates: `x0, ..., x8`.\n",
    "We will standardize the 5 numerical covariates, and leave the binary covariates as is.\n",
    "Note that PyTorch require variables of type `'float32'`.\n",
    "\n",
    "We like using the `sklearn_pandas.DataFrameMapper` to make feature mappers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols_standardize = ['x0', 'x1', 'x2', 'x3', 'x8']\n",
    "cols_leave = ['x4', 'x5', 'x6', 'x7']\n",
    "\n",
    "standardize = [([col], StandardScaler()) for col in cols_standardize]\n",
    "leave = [(col, None) for col in cols_leave]\n",
    "\n",
    "x_mapper = DataFrameMapper(standardize + leave)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_mapper.fit_transform(df_train).astype('float32')\n",
    "x_val = x_mapper.transform(df_val).astype('float32')\n",
    "x_test = x_mapper.transform(df_test).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label transforms\n",
    "\n",
    "The survival methods require individual label transforms, so we have included a proposed `label_transform` for each method.\n",
    "In this case `label_transform` is just a shorthand for the class `pycox.preprocessing.label_transforms.LabTransDiscreteTime`.\n",
    "\n",
    "DeepHit is a discrete-time method, meaning it requires discretization of the event times to be applied to continuous-time data.\n",
    "We let `num_durations` define the size of this (equidistant) discretization grid, meaning our network will have `num_durations` output nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_durations = 10\n",
    "labtrans = DeepHitSingle.label_transform(num_durations)\n",
    "get_target = lambda df: (df['duration'].values, df['event'].values)\n",
    "y_train = labtrans.fit_transform(*get_target(df_train))\n",
    "y_val = labtrans.transform(*get_target(df_val))\n",
    "\n",
    "train = (x_train, y_train)\n",
    "val = (x_val, y_val)\n",
    "\n",
    "# We don't need to transform the test labels\n",
    "durations_test, events_test = get_target(df_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pycox.preprocessing.label_transforms.LabTransDiscreteTime"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(labtrans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural net\n",
    "\n",
    "We make a neural net with `torch`.\n",
    "For simple network structures, we can use the `MLPVanilla` provided by `torchtuples`.\n",
    "For building more advanced network architectures, see for example [the tutorials by PyTroch](https://pytorch.org/tutorials/).\n",
    "\n",
    "The following net is an MLP with two hidden layers (with 32 nodes each), ReLU activations, and `num_nodes` output nodes.\n",
    "We also have batch normalization and dropout between the layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_features = x_train.shape[1]\n",
    "num_nodes = [32, 32]\n",
    "out_features = labtrans.out_features\n",
    "batch_norm = True\n",
    "dropout = 0.1\n",
    "\n",
    "net = tt.practical.MLPVanilla(in_features, num_nodes, out_features, batch_norm, dropout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you instead want to build this network with `torch` you can uncomment the following code.\n",
    "It is essentially equivalent to the `MLPVanilla`, but without the `torch.nn.init.kaiming_normal_` weight initialization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# net = torch.nn.Sequential(\n",
    "#     torch.nn.Linear(in_features, 32),\n",
    "#     torch.nn.ReLU(),\n",
    "#     torch.nn.BatchNorm1d(32),\n",
    "#     torch.nn.Dropout(0.1),\n",
    "    \n",
    "#     torch.nn.Linear(32, 32),\n",
    "#     torch.nn.ReLU(),\n",
    "#     torch.nn.BatchNorm1d(32),\n",
    "#     torch.nn.Dropout(0.1),\n",
    "    \n",
    "#     torch.nn.Linear(32, out_features)\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the model\n",
    "\n",
    "To train the model we need to define an optimizer. You can choose any `torch.optim` optimizer, but here we instead use one from `tt.optim` as it has some added functionality.\n",
    "We use the `Adam` optimizer, but instead of choosing a learning rate, we will use the scheme proposed by [Smith 2017](https://arxiv.org/pdf/1506.01186.pdf) to find a suitable learning rate with `model.lr_finder`. See [this post](https://towardsdatascience.com/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6) for an explanation.\n",
    "\n",
    "We also set `duration_index` which connects the output nodes of the network the the discretization times. This is only useful for prediction and does not affect the training procedure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DeepHit has a loss that is a combination of a negative log-likelihood and a ranking loss. `alpha` is a parameter that controls the linear combination between the two, and `sigma` is a parameter used by the ranking loss. `alpha = 1` give a loss only containing the negative log-likelihood and `alpha = 0` give a pure ranking loss. Note that this is different than the original paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = DeepHitSingle(net, tt.optim.Adam, alpha=0.2, sigma=0.1, duration_index=labtrans.cuts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEKCAYAAADTgGjXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhU5fXA8e+ZrCSQEEjYt7Dva9hBRKsCWrBqKVBRUFGrqLVVi62tLf1ZbbVaEVxAUUtF3BUVRVQUZRHCTkAg7IGwQ1iSkO38/pgJhpBlMpnJZJLzeZ55zL1z733PjDHHdxdVxRhjjPGEw98BGGOMCVyWRIwxxnjMkogxxhiPWRIxxhjjMUsixhhjPGZJxBhjjMeC/R2AN8XGxmqLFi38HYYxxgSU1atXH1XVOE/urVJJpEWLFiQmJvo7DGOMCSgissfTe605yxhjjMcsiRhjjPGYJRFjjDEeq1J9IsaYqic7O5uUlBQyMzP9HUrACw8Pp0mTJoSEhHjtmZZEjDGVWkpKCrVq1aJFixaIiL/DCViqyrFjx0hJSSE+Pt5rz7XmLGNMpZaZmUndunUtgZSTiFC3bl2v1+iqTRI5euYcqWkZ/g7DGOMBSyDe4YvvsdokkUc+2MSNL/+A7Z9ijDHeU22SyK6jZ9lx5CzJh8/4OxRjTIA5efIkzz//fJnvGzFiBCdPnizzfRMmTODdd98t833+UG2SSH5T1hebD/k5EmNMoCkuieTm5pZ434IFC6hdu7avwqoUqsXorLPncjiVmQM4k8jdQ1v7OSJjjCf+9nESmw+c8uozOzaK4tGfdyrxmilTprBjxw66d+9OSEgINWvWpGHDhqxbt47Nmzdz7bXXsm/fPjIzM7nvvvu4/fbbgZ+WYjpz5gzDhw9n0KBBLFu2jMaNG/PRRx9Ro0aNUuP76quveOCBB8jJyaF379688MILhIWFMWXKFObPn09wcDBXXnklTz31FO+88w5/+9vfCAoKIjo6miVLlnjlOyqJz2siIjJMRLaKSLKITCni/WdEZJ3rtU1EThZ4r5mIfCEiW0Rks4i08CSG1DTnaIR29Wuxft9JDqbZeHNjjPueeOIJWrVqxbp163jyySdZuXIljz32GJs3bwZg9uzZrF69msTERKZNm8axY8cuesb27du5++67SUpKonbt2rz33nullpuZmcmECRN466232LhxIzk5ObzwwgscP36cDz74gKSkJDZs2MAjjzwCwNSpU1m4cCHr169n/vz53v0SiuHTmoiIBAEzgCuAFGCViMxX1c3516jq/QWuvwfoUeAR/wUeU9VFIlITyPMkjvymrJsGNOdPH2xi0ZZDjO/X3JNHGWP8qLQaQ0Xp06fPBXMtpk2bxgcffADAvn372L59O3Xr1r3gnvj4eLp37w5Ar1692L17d6nlbN26lfj4eNq2bQvAzTffzIwZM5g8eTLh4eHcdtttXH311VxzzTUADBw4kAkTJjB69Giuu+46b3zUUvm6JtIHSFbVnaqaBcwDRpVw/VjgTQAR6QgEq+oiAFU9o6rpngSRetJZ87ikTRwt6kawyPpFjDHlEBkZef7nb775hi+//JLly5ezfv16evToUeRcjLCwsPM/BwUFkZOTU2o5xY0mDQ4OZuXKlVx//fV8+OGHDBs2DIAXX3yR//u//2Pfvn107969yBqRt/k6iTQG9hU4TnGdu4iINAfiga9dp9oCJ0XkfRFZKyJPumo2he+7XUQSRSTxyJEjRQaRmpaJCNSPCufKTg1YvuMopzKzy/O5jDHVSK1atTh9+nSR76WlpRETE0NERAQ//vgjK1as8Fq57du3Z/fu3SQnJwMwZ84chgwZwpkzZ0hLS2PEiBH85z//Yd26dQDs2LGDvn37MnXqVGJjY9m3b19Jj/cKX3esFzWzpbiJGmOAd1U1f7hDMDAYZ/PWXuAtYALwygUPU50JzARISEgo8tmpaRnE1gwjNNjBFR3rM3PJTr7deoSfd2tU6gfIy1N2Hj3DvhMZJDSPoVa499acMcYEhrp16zJw4EA6d+5MjRo1qF+//vn3hg0bxosvvkjXrl1p164d/fr181q54eHhvPrqq/zyl78837F+5513cvz4cUaNGkVmZiaqyjPPPAPAgw8+yPbt21FVLr/8crp16+a1WIojvpx8JyL9gb+q6lWu44cBVPXxIq5dC9ytqstcx/2AJ1T1UtfxeKCfqt5dXHkJCQla1KZUN81eycn0LOZPHkRuntLnsS8Z0DqW58b2KOIpkJunvLRkB0u2HWHT/lOcOeesdoYGOxjaLo5rujbi8g71iAitFoPbjPGrLVu20KFDB3+HUWUU9X2KyGpVTfDkeb7+K7gKaCMi8cB+nLWNcYUvEpF2QAywvNC9MSISp6pHgMsAj7YtPJiWQXyssw0zyCFc3qEen208SFZOHqHBF7fofbvtMP/6fCsdGkZxbY9GdG1Sm4bR4Xy15TCfbkxlYdIh6keF8cVvhxAdYTUTY0z15dMkoqo5IjIZWAgEAbNVNUlEpgKJqpo/Bm0sME8LVItUNVdEHgC+EueCL6uBWZ7EkXoykwGtYs8fX9mxAW8nprBi5zEuaXvxtsKfbjhIVHgwH9098IIkM7hNHH++piOLNh/izv+tZt6qvdwxpJUnIRljDHfffTdLly694Nx9993HxIkT/RRR2fm8PUZVFwALCp37S6HjvxZz7yKga3nKP52ZzelzOTSMDj9/blCbWGqEBPHZpoMXJZFzObl8sfkgV3VqUGQtJcghDOvcgP4t6/Last3cMiiekKBqM/HfGL9Q1Sq5COOMGTMqtDxfdF9U+b9++RMLG9b+aWZoeEgQw7s0YP66/ReN0lqafJTTmTlc3aVhic+9bXA8qWmZLNiY6v2gjTHnhYeHc+zYMVs8tZzy9xMJDw8v/eIyqPI9wwfyk0j0hV/cxAHxvL9mP2+v2sdtg1ueP//JhlSiwoMZ2DqWkgxtV4+WsZG88v0uRnZrVCX/L8mYyqBJkyakpKRQ3BB+4778nQ29qconkdSTztnqhZNIlybRJDSP4b/L9zBxYDxBDuFcTi6LNh8qtimrIIdDmDgonj9/uInEPSfo3aKOzz6DMdVZSEiIV3fiM95VpZqzsnIuXhWl4ETDwiYOjGfv8XS+/vEwAN9vdzVldS25KSvf9T0bUzsihJe/21m+wI0xJkBVqSSy9dBpDp++cLmB1LQM4mqGFdn5fVWn+jSMDufVpbsA+HSjqymrVclNWfkiQoMZ16cZX2w+xJ5jZ8v/AYwxJsBUqSQCsGbPhRvApKZlXtCpXlBwkIPx/ZuzbMcxNqaksSjJvaasgm4e0IJgh/Dq0t3lCdsYYwJSlUoiAqzde+KCc6lpmTQsoikr39jezQgPcXDfvLWcPpfDCDebsvLVjwpnZLfGzP1h70VlFyctI5u8PBtpYowJfFUqidQIDWJNgT/kqkrqyQwa1i4+icREhvKLHo3ZefQs0TVC3G7KKuiRqztQLyqM3/xvDUdOnyvx2uTDp+n7jy8Z/dJydhyxrXqNMYGtSiWRiNBg1qekne9gP30uh7NZuTSKLnn3sAkDnCM/ruxYv0xNWfliIkN5aXwvTqRnMXnuGrJzi972RFV55MNNhAQ52H74DMOf/Y7nv0kmp5jrjTGmsqtiSSSIrJw8Nqc6t8/M30ekQXTJk2vaNajFS+N78cBV7Twuu1OjaJ64vgs/7DrO4wt+LPKaD9ftZ8XO40wZ3p5Fv7uEy9vX41+fb+Xa55fy6YZUzuWUvF+zMcZUNlVqnkhEaBBpwJo9J+jetDYHXDsaNiqhOSvfVZ0alLv8X/Rowvp9acxeuov2DWoxunfT8++lpWfz2Kdb6Na0NmN7N8PhEF64sRcLNqby2KdbuHvuGqJrhDCqeyN+1bspnRpFF1nG4dOZnMnMoWVczXLHa4wx5VWlaiIhQQ4aRYef7xc5v+RJKc1Z3vSnqzvQv2VdHnpvA79/e/35ZVWe+mIrx89m8di1nXE4fprdPqJLQ5Y8NJQ5t/ZhSNs45q3ax8jpS9m0P+2iZ6sqt76WyLD/fMeXtjujMaYSqFJJBKBH8xjW7HEmkdSTGTgE6tUKK+Uu7wkJcvD6LX2YPLQ1H6xNYdgzS3h16S7+98Meburfgs6NL65hBDmEwW3imDa2B8umXEZEaBDTv06+6Lpvtx1h4/40omoEc+f/VvPx+gMV8ZGMMaZYVS6J9GoWw4G0TA6mZXIgLZN6tcIJruBVdkODHTxwVTve+80AwkOC+NvHm4mtGcbvrmxb6r2xNcOYOKAFnycdZOvBC7fjfH7xDhpGh7Po/iH0bB7DvfPW8vYq329/aYwxxalySaRn8xgA1uw9wcG0zFI71X2pR7MYPr13ML+/oi3Tx/Ygys2tdW8ZFE9kaBDTF/9UG1m56zgrdx/n9ktaEhMZyusT+zC4TRwPvbeBl77dYSucGmP8osolkY4NowgLdrBmzwkOpGW41anuSzVCg7jn8jb0bVnX7XtqR4Qyvn8LPtlw4Pxckue/SaZOZChjejc7/9xZN/Xi6i4NefyzH5k8d+35bXyNMaaiVLkkEhrsoEvjaFbvPUHqycwK7VT3ptsGxxMW7GDG4mQ27U/jm61HuHVQPDVCg85fExYcxPRxPZgyvD2fbUpl1PTvST58uoSnGmOMd1W5JALQq3kMG1LSyMjOvWgJ+EARWzOMcX2a89G6A/x1fhK1woIZ37/5RdeJCHcOacX/butLWkY2I6cv5dtttu+CMaZiVMkk0qNZDLmutakCtSYCcMeQlgQ5hMQ9J7hpQPMS+1QGtIrlk3sG07xuJJPfWMNOW1LFGFMBqmQS6dm89vmfS1o3q7KrHxXOuD7NqBkWzMSBpW/K0yA6nFk39SIk2MHtc1ZbH4kxxueqZBKpVyucpnWcNZBAbc7K96erO7D4gUuJreneXJcmMRFMH9uDnUfO8MDb623UljHGp6pkEgHo2SyGIIcQ5+Yf38oqJMhBXBknSw5oHcsfR3Tg86SDPP/NDh9FZowxFbB2logMA54FgoCXVfWJQu8/Awx1HUYA9VS1doH3o4AtwAeqOtndcu+6tDUDW8dW+ETDyuLWQfFsSEnjqS+2IuI8DgsOKv1GY4wpA58mEREJAmYAVwApwCoRma+qm/OvUdX7C1x/D9Cj0GP+Dnxb1rLbNahFuwa1PIq7KhAR/nl9VzKyc/nX51t5JzGFv/y8I0Pb1fN3aMaYKsTX/5veB0hW1Z2qmgXMA0aVcP1Y4M38AxHpBdQHvvBplFWUc0JiAq9O7I0AE19dxW2vryI9yzrcjTHe4esk0hgouLhTiuvcRUSkORAPfO06dgD/Bh4sqQARuV1EEkUk8cgRmx9RlKHt6vH5by/hoWHt+HLLYd5YsdffIRljqghfJxEp4lxxw4XGAO+qav7OTHcBC1S1xBUGVXWmqiaoakJcXFw5Qq3aQoMd3HVpa/q1rMOrS3cVu/uiMcaUha+TSArQtMBxE6C49cvHUKApC+gPTBaR3cBTwE0i8kRRNxr3TRrckgNpmSzYmOrvUALSsTPn+Gjdfs7aHBxjAN+PzloFtBGReGA/zkQxrvBFItIOiAGW559T1V8XeH8CkKCqU3wcb5U3tF09WsVFMnPJTkZ2a4RIUZXFipc/n6WkeHLzlC2pp/hh13H2HU/ntsHxNImJqKgQUVXueXMty3YcIyo8mLF9mnHTgBY0rh24qyIYU14+TSKqmiMik4GFOIf4zlbVJBGZCiSq6nzXpWOBeWoz43zO4RBuG9ySh9/fyPKdxxjQKtYrzz18KpP56w/Qr2XdIjfeKk5envLxhgM8++V24mqFMX1cz4vmxaScSOfvn2xmWfIxTrtqAEEOYcHGVF6/pQ8dGkZ55TOUZu7KvSzbcYw7hrQk5XgGs77bycvf72JI2zgSWsTQs1kMXZtEExFapXadNqZEUpX+bickJGhiYqK/w6j0MrNzGfTPr+nSOJpXJ/Yp97Ne+X4Xzy9O5myWsztrcJtY7rikFQNb1yUzO49NB9JYt/ckB09l0rxuBC1ja9IyLpJN+9P49xfb2HroNG3r12Tv8XTqRIQy6+aE83vMf5F0kAfeWY8qjOzeiD7xdegTX4dTGTncPHslZ7NymHVTAv3KsNS+J/afzOCqZ5bQtUk0b9zWFxEh5UQ6c5bvYdHmQ+w8ehZwJrdhnRrw+PVd3N4/xhh/E5HVqprg0b2WRKqnaV9t5+lF21h0/yW0qV/8fBpVZUvqaT7blMoXSYfIU6VlXCSt4mpSt2YYry3bxb7jGVzZsT73Xt6G77YfZfbSXRw5fY4GUeEcOXPu/GKYYcEOzuVc2KEfHxvJ765oy9VdGpJ04BS3z0nkZHo2T1zfhfX70pi9dBddGkczfVwPmteNvODe/SczuOmVH9h3IoNnf9Wd4V0aev+Lcn0HN81eyeo9J1j420toWufiJrQTZ7NYu+8Ey5KP8dqy3TStE8FL43vRtoTv1pjKwpKIiyUR9x0/m8WAJ75iVLfG/POGrhe8l584Pt14gAUbD7Lr6FkcAn3j61IrPJgdR86w51g6OXlK2/o1+cs1nRjU5qdmsXM5uXy09gBf/3iY1vVq0r1pbbo1rU1szVAOnTrHziNn2HH0LFHhwVzdpeEFqwocPp3JnXNWs2bvSQAmDGjBwyPaFzvb/sTZLG55fRVr954koXkM4/s3Z3jnhoQGe2/MyNur9vHQexv428hO3DygRanXr9x1nLveWEN6Vg5P3tCNq7v6JrkZ4y0+TyKuORvdgEZABpCkqoc8KdCXLImUzSMfbuR/K/bSrE4EHRtG0alRFNl5yqcbDrDjyFmCHEL/lnUZ0aUhV3WqT90C65Bl5+ZxMC2ThtHe38P+XE4uMxbvoEvjaK7oWL/U6zOycnnjhz3MWbGHPcfSia0ZxvW9GjOkTRw9m8cQHuLZci+5ecriHw9z/9vr6NAwinmT+uFwuDcQ4dCpTH7zP2cyvPey1tx/RdtKM4jBmMJ8lkREpBXwB+BnwHbgCBAOtAXSgZeA11W1Ukw6sCRSNmfO5fDf5btJ2n+KpANp7D6Wjgj0ja/DNV0bMbxzgwsSR2WXl6d8l3yUOct3s3jrEXLzlLBgB71b1KFHs9o0jYmgSZ0aNI2JoHZECMEOB8FBQrBDUIWcPCU3TzmensX7q1N4c+VeDrgS5ZuT+tEiNrLUGArKysnjkQ838nZiCjf3b86jP+/kdhIypiL5Mom8CbwAfFd45JSI1MM5XPeEqr7uSeHeZkmkfM6cyyE7J4+YyFB/h1JupzOzWbnrOEuTj7E0+SjbD58mr4wtt4NaxzKubzOu6FifEA9rW6rKPxZsYdZ3u7iuZ2P+dX3XarsoqKm8rE/ExZKIKU5WTh6paRmknMhg3/F0TmfmkJOn5OTmkZ2nOASCHUKQw0FosIPL2tcjvow1j+KoKtO/Tubfi7ZxZcf6PDeuh62obCqV8iSREge0i8h1Jb2vqu97UqgxFS002EHzupEXjfCqCCLCPZe3oVZ4MH/9eDN3v7GWF27s6XHtxpjKpLRZUT93/bMeMADX4og49//4BrAkYoybJgyMJ8gh/PmjJB54Zz3PjO5ufSQm4JWYRFR1IoCIfAJ0VNVU13FDnPuEGGPKYHz/Fpw+l8O/Pt9KZFgwj13b2UZtmYDm7voMLfITiMshnCO0jDFldNelrTmdmcML3+ygZlgwD17Vzpq2TMByN4l8IyILca6yqzgXUlzss6iMqeIeuqodZzJzmLlkJ/9dvpuuTWrTo1ltejevw+C2sdbxbgKG26OzROQXwCWuwyWq+oHPovKQjc4ygSQvT1mYdJCVu4+zdu9Jkg6kkZ2rxESEcG2Pxvyqd1PaN6iYxSVN9VYhQ3xdOw+2UdUvRSQCCFLV054U6iuWREwgy8zOZcXOY7yzOoVFSYfIys2jTb2atG8YRSvXemW9msfQyJaeN17msyG+BQqYBNwO1AFa4dzi9kXgck8KNcZcLDwkiEvb1ePSdvU4cTaLD9buZ8n2I6zfd5JPNhxAFUKDHNx5aSvuurSVx8u5GONN7q6dtQ7oA/ygqj1c5zaqahcfx1cmVhMxVVVmdi47jpxh5pKdfLTuAE3r1OBvIztxWfvS1xYzpjTlqYm4OyTknKpmFSgwmOL3SjfGeFl4SBCdGkXz7JgezJ3Ul7DgIG55LZE756zm8KlMf4dnqjF3k8i3IvJHoIaIXAG8A3zsu7CMMcUZ0CqWBfcO5qFh7Vi89TA/e/pb3k7cR1VawsgEDneTyBScK/huBO4AFgCP+CooY0zJQoMd3HVpaz67bzDtG0Tx0LsbuGn2SlJOpPs7NFPN2AKMxgS4vDzljZV7eWLBFkSEP1/TgdEJTW0mvHGbz/tERGSgiCwSkW0islNEdonITk8KNMZ4l8MhjO/XnM9/ewmdG0fxh/c2ctvridZXYiqEu81ZrwBPA4OA3kCC65/GmEqiaZ0I5t7Wj0d/3pHvk49y5X+WsGzHUX+HZao4d5NImqp+pqqHVfVY/sunkRljyszhECYOjGfBfYOJqxnGra8lsmKn/adqfKfEJCIiPUWkJ7BYRJ4Ukf7551znSyUiw0Rkq4gki8iUIt5/RkTWuV7bROSk63x3EVkuIkkiskFEfuXRJzSmGmoVV5O5k/rROKYGt7y2ipW7jvs7JFNFlbY9bkmLLKqqXlbiw0WCgG3AFUAKsAoYq6qbi7n+HqCHqt4iIm1dZWwXkUbAaqCDqp4srjzrWDfmQodPZzJm5goOpWXy31v70Kt5HX+HZCohny17oqpDXQW0VNULOtJFpKUbz+8DJOffKyLzgFFAkUkEGAs86ip7W4E4DojIYSAOKDaJGGMuVK9WOG9O6seYmSu48eWVXNI2lp7NYujZPIYujaNt6RRTbu4uBf8uULj56h2gVyn3NQb2FThOAfoWdaFrgcd4fto9seB7fYBQYEcR792Oc10vmjVrVko4xlQ/9aOcieRfC38kcfcJFiYdAqBGSBA39GrChIEtaBVX089RmkBV2h7r7YFOQHSh/dajgHA3nl/UQPXi2s/GAO+qam6hGBoCc4CbVTXvooepzgRmgrM5y42YjKl2GkSH8/To7gAcOX2OtXtP8MXmQ7y1ah9zVuxhaLs47hramt4trLnLlE1pNZF2wDVAbX7abx3gNDDJjeenAE0LHDcBDhRz7Rjg7oInRCQK+BR4RFVXuFGeMaYUcbXCuLJTA67s1IA/DGvP3B/2MmfFHsbNWsEbt/WjT7wlEuM+d1fx7a+qy8v8cOdCjdtwLhm/H2fH+jhVTSp0XTtgIRCvroBEJBT4DPhYVf/jTnnWsW6MZ9IysvnFjKWkZWTz0eSBNImJ8HdIpgJVxCq++0TkAxE5LCKHROQ9EWlS2k2qmgNMxpkgtgBvq2qSiEwVkZEFLh0LzNMLM9ponDspTigwBLi7m/EaY8ogukYIs25OICsnj9v/u5r0rBx/h2QChLs1kUXAXJx9EwA3Ar9W1St8GFuZWU3EmPJZ/ONhbnl9FSM6N2T6uB62/lY1URE1kXqq+qqq5rher+EcbmuMqUKGtq/HlGHt+XRjKs8s2mbLy5tSuTvE94iI3Ai86ToeC9haCsZUQbdf0pLkw2eY9nUyGdm5PDy8Aw6H1UhM0dxNIrcA04FnXMdLXeeMMVWMiPDP67tSIzSIWd/t4vjZbJ64vgshQe42XJjqxK0koqp7gZGlXmiMqRIcDuFvIztRJzKU/3y5nZPpWcz4dU+b4W4u4u5+Ik08GZ1ljAlcIsJvf9aWv4/qxNdbD3PDi8vYffSsv8MylYy79dNXgflAI5xLmXzsOmeMqeLG92/BrPEJ7DuewTXPfc8nG4qbL2yqI3eTSJyNzjKm+vpZx/p8eu8g2tavyeS5a/njBxvJzM4t/UZT5bmbRI6KyI0iEuR63YiNzjKmWmkSE8Fbd/TnziGtmPvDXm57PdESiXE7idyCcwb5QSAVuAEbnWVMtRMS5GDK8PY89ctuLN1xlDvmrOZcjiWS6sxGZxljyuyGXk3Iyc1jyvsbufuNNTz/616EBtsQ4OrIrSQiIvHAPUCLgveoqiUWY6qpMX2akZ2n/PnDTdzz5hqmj+tpc0mqIXcnG34IvIJzVNZFe3oYY6qn8f2ak52Tx9RPNvPK97u4c0grf4dkKpi7SSRTVaf5NBJjTEC6ZVA8S5OPMv3rZK7v2YS4WmH+DslUIHfrns+KyKMi0l9Eeua/fBqZMSZg/OnqDpzLyeWphVv9HYqpYO7WRLoA44HL+Kk5S13HxphqrmVcTSYMaMHL3+9ifP/mdG4c7e+QTAVxtybyC6Clqg5R1aGulyUQY8x591zehjoRofzt4yRbQr4acTeJrMe5z7oxxhQpKjyEB65qx6rdJ/hkQ6q/wzEVxN0kUh/4UUQWisj8/JcvAzPGBJ7RCU3p0DCKxxds4ew522K3OnC3T+RRn0ZhjKkSghzC1FGd+NVLy3novQ1MH2tb7FZ17tZEEoHvVPVbnMueRAPLfBaVMSZg9W5RhweuasenG1J55ftd/g7H+Ji7SWQJEC4ijYGvgInAa74KyhgT2H4zpBVXdarP45/9yIqdtlZrVeZuEhFVTQeuA55T1V8Andy6UWSYiGwVkWQRmVLE+8+IyDrXa5uInCzw3s0ist31utnNWI0xfiYiPPXLbjSvG8HkuWs4mJbp75CMj7idRESkP/Br4FPXuVL3yRSRIGAGMBzoCIwVkY4Fr1HV+1W1u6p2B54D3nfdWwdnX0xfoA/wqIjEuBmvMcbPaoWH8NKNvUjPyuXO/63mjHW0V0nuJpH7gIeBD1Q1SURaAovduK8PkKyqO1U1C5gHjCrh+rHAm66frwIWqepxVT0BLAKGuRmvMaYSaFO/Fk+P7s7G/WmMf+UH0tKz/R2S8TK3koiqLlHVkar6T9fxTlW9141bGwP7ChynuM5dRESaA/HA12W91xhTeQ3r3IDnf92TpP2nGDtrBcfOnPN3SMaLSkwiIjJTRLoU816kiNwiIr8u6RFFnCtuKusY4F1Vzd/hxq17ReR2EUkUkcQjR46UEIoxxl+u6tSAWTcnsPPoGX41c4X1kVQhpdVEnlYJkywAABjsSURBVAf+LCJbROQdEXleRGaLyHc4h/jWAt4t4f4UoGmB4ybAgWKuHcNPTVlu36uqM1U1QVUT4uJs23djKqshbeN4fWIfUk9mcOMrP9jWulVEiUlEVdep6migN84O8u+A+cBtqtpNVZ9V1ZLqpquANiISLyKhOBPFRTPdRaQdEAMsL3B6IXCliMS4OtSvdJ0zxgSovi3r8vyNvUg+fIZpX233dzjGC9zdHvcM8E1ZH66qOSIyGecf/yBgtqtjfiqQqKr5CWUsME8LrNqmqsdF5O84ExHAVFU9XtYYjDGVy5C2cdzQqwkvLdnJiC4NbcXfACdVabXNhIQETUxM9HcYxphSpKVn87NnviWuZhgfTR5o2+r6mYisVtUET+61f3PGmAoXHRHC30d1YnPqKWYu2envcKq1pxdtK9f9ZU4iIuIQkahylWqMqfaGdW7IiC4NePar7SQfPuPvcKqt/ScyynW/W0lEROaKSJSIRAKbga0i8mC5SjbGVHt/HdmJGiFB/PattWRk2Wgtf8jILt9KAu7WRDqq6ingWmAB0AzndrnGGOOxerXCeXp0N5IOnOKBd9fbjoh+kF7O5O1uEgkRkRCcSeQjVc2m+EmDxhjjtss71OcPw9rz6YZUnvs62d/hVDsVlUReAnYDkcAS1xIlp8pVsjHGuNxxSUuu69GYpxdt4/NNtrVuRSpvM6K7a2dNU9XGqjpCnfYAQ8tVsjHGuIgI/7iuC92b1ub+t9azaX+av0OqNtKzKqBPRETuc3Wsi4i8IiJrgMvKVbIxxhQQHhLEzPG9qB0Rws2zV7Lt0Gl/h1QtVEhNBLjF1bF+JRCHc2fDJ8pVsjHGFFIvKpw3butLkEMYN+sHG/pbAdLLuYaZ25tSuf45AnhVVddT9Cq7xhhTLi3jajJ3Uj8Axs1awc4jlkh8qaI61leLyBc4k8hCEakF5JWrZGOMKUbrejV5c1JfcvOUcbN+IOVEur9DqpJy85SsnPL9KXc3idwKTAF6u/ZaD8XZpGWMMT7Rpn4t3pjUl9OZ2fzfJ1v8HU6VVN5OdXB/dFYezv08HhGRp4ABqrqh3KUbY0wJ2jeI4o4hrfg86SCJu20Rb2/zxioB7o7OegLnPuubXa97ReTxcpdujDGluG1wPPVqhfGPBVtsRruXlbc/BNxvzhoBXKGqs1V1NjAMuLrcpRtjTCkiQoP53RVtWbP3JJ9vOujvcKqUikwiALUL/Gy7yBhjKswvE5rStn5N/vn5j+XuCDY/Ke/ii+B+EnkcWCsir4nI68Bq4B/lLt0YY9wQ5BAeHt6B3cfSmfvDHn+HU2VUWE1EVd8E+gHvu179VXVeuUs3xhg3Xdoujv4t6zLt62SOnjnn73CqBJ8nERHpmf8CGgIpwD6gkeucMcZUCBHhkWs6cPZcDte/sIxdR8/6O6SA543RWcGlvP/vEt5TbP0sY0wF6tQomrmT+jHpv4lc9/xSXr45gV7N6/g7rIDl85qIqg4t4XU+gYjIFeWOxBhj3NCreQzv/2YA0TVCGDvrBxZstKXjPVVhkw3d8E8vPccYY0rVIjaS9+8aSOdGUdw9dw1vrtzr75ACUoVNNnRDsYsxisgwEdkqIskiMqWYa0aLyGYRSRKRuQXO/8t1bouITBMRW/TRGANAnchQ5k7qx5C2cTz8/kZmf7/L3yEFnPTsXIId5fuz6q0kUuQ0UhEJAmYAw4GOwFgR6VjomjbAw8BAVe0E/NZ1fgAwEOgKdAZ6A0O8FK8xpgoIDwnipfG9GNapAVM/2cyMxba9bllkZOVSIzSoXM/wVhIpTh8gWVV3qmoWMA8YVeiaScAMVT0BoKqHXecVCMe52GMYEAIc8nG8xpgAExYcxPRxPfhFj8Y8uXArT3+x1d8hBYz0rBwiKkkS2V3M+cY4hwTnS3GdK6gt0FZElorIChEZBqCqy4HFQKrrtVBVbSlPY8xFgoMc/PuX3bihVxOmfZ3MhpST/g4pIKRn5RIRWtog3ZK5nUREZICIjBORm/Jf+e+p6nXF3VbEucJNX8FAG+BSYCzwsojUFpHWQAecqwc3Bi4TkUuKiOt2EUkUkcQjR464+3GMMVWMwyE8+vOO1I0MtcUa3ZSRlUuNkAqoiYjIHOApYBDOvoneQIIbt6YATQscNwEOFHHNR6qaraq7gK04k8ovgBWqekZVzwCf4Zw1fwFVnamqCaqaEBcX587HMcZUUbXCQ7jvZ21YsfM432y1/6ksjbMmUjHNWQk4O77vUtV7XK973bhvFdBGROJFJBQYA8wvdM2HwFAAEYnF2by1E9gLDBGRYBEJwdmpbs1ZxpgSje3TjPjYSB7/bAs5ubZYY0nSsyuuY30T0KCsD1fVHGAysBBnAnhbVZNEZKqIjHRdthA4JiKbcfaBPKiqx4B3gR3ARmA9sF5VPy5rDMaY6iUkyMFDV7Vj26EzvLcmxd/hVGoZXuhYL7FHRUQ+xtmHUQvYLCIrgfMrn6nqyOLuLXDNAmBBoXN/KfCzAr9zvQpekwvcUfpHMMaYCw3r3ICezWrz9KJt/Lxbo3J3HldV3uhYL+3up8r1dGOM8QMR4Y8jOnDDi8uZtWQX9/2sjb9DqpS8MU+kxCSiqt8CiEg8kKqqma7jGkD9cpVsjDE+lNCiDld3bcizX22jRWwEo7oXnl1g0rNyiaiI0VnAO0DBHqpc1zljjKm0nrqhG33i63D/W+v4eH3hgaHVW16ekpFdcaOzgl0zzgFw/RxarpKNMcbHaoQG8crNvUloXoffvrWOz2zF3/Myc5yLL9aooMmGRwqMpkJERgFHy1WyMcZUgMiwYGZP7E33prW55821fLXFVk+Cn/YSqaiayJ3AH0Vkr4jsA/6AjZwyxgSImmHBvDaxN+0b1uL376zn8OlMf4fkd/nLwFfIPBFV3aGq/XCuxNtRVQeoqi2XaYwJGLXCQ3h2TA8ysnL54/ubqv2yKN6qibjdGCYiVwOdgPD8bT1UdWq5SjfGmArUKq4mD17Vjv/7dAsfrN3PdT2b+Dskv8nf1bBCmrNE5EXgV8A9OBdV/CXQvFwlG2OMH0wcGE/vFjE8Oj+Jg2nVt1nrfHNWSMV0rA9Q1ZuAE6r6N6A/Fy6saIwxASHIITx5QzdycpUp72+ots1aFd2xnpFfrog0ArKB+HKVbIwxftIiNpIpw9vzzdYjvLZst7/D8Yv07IpNIp+ISG3gSWANzk2o3ixXycYY40fj+zXnio71mfrJ5mo5ETHD1SdSUaOz/q6qJ1X1PZx9Ie0LLqJojDGBxuEQnhvbg94t6vC7t9exZFv12n/kp+asCugTEZFwEfmdiLwPzAVuEZHwcpVsjDF+Fh4SxMs3J9C6Xi3umLOaNXtP+DukClPRfSL/xTm89zlgOs5ta+eUq2RjjKkEosJDeP2W3tSLCuOW11bx48FT/g6pQmRk5SICYcFu75JeJHfvbqeqt6rqYtfrdpw7EBpjTMCrVyuc/93al/DgIMbMXMGm/Wn+Dsnn8lfwzZ/35yl3k8haETm/v7mI9AWWlqtkY4ypRJrWieDtO/oTGRrM2FkrWFvFm7YysnPKvfgilJJERGSjiGwA+gLLRGS3iOwClgOXlLt0Y4ypRJrVjeDtO/tTJzKUG1/+gZW7jvs7JJ9x7mpYvv4QKL0mcg3wc2AYznkhQ4BLXT9fXe7SjTGmkmlcuwZv39GfBtHh3Dx7Jct3HPN3SD5RIUlEVfeU9Cp36cYYUwnVjwpn3u39aRJTg4mvrWTZjqq384U3tsYF9/tEjDGmWomrFcabt/ejWZ0IbnltFcuSq1YiSc/KqZDmLGOMqbZia4Yxd1I/mteJ5JbXq1YiSc/KLffii1ABSUREhonIVhFJFpEpxVwzWkQ2i0iSiMwtcL6ZiHwhIltc77fwdbzGGFNQbM0w3pjUl+Z1Irlp9koefn8D+09mlH5jJeeN/dWhDPuJeEJEgoAZwBVACrBKROar6uYC17QBHgYGquoJEalX4BH/BR5T1UUiUhPI82W8xhhTlNiaYcy7vR/PfLmNeSv38d7q/Yzp05TJQ1tTLyowF++oqNFZ5dUHSFbVnaqaBcwDRhW6ZhIwQ1VPAKjqYQAR6QgEq+oi1/kzqpru43iNMaZIMZGhTB3VmcUPXsr1vZow94e9jJj2XcDWSgKlY70xsK/AcYrrXEFtgbYislREVojIsALnT4rI+yKyVkSedNVsLiAit4tIoogkHjlSvRZQM8ZUvMa1a/D4dV345N5BnMvOY9Lried3CQwUqhowHetFzacvvANMMNAG5/yTscDLrmXng4HBwANAb6AlMOGih6nOVNUEVU2Ii4vzXuTGGFOC9g2imDauB1sOnuLBdwJrc6tzOXnkaflX8AXfJ5EULtwBsQlQeOH+FOAjVc1W1V3AVpxJJQVY62oKywE+BHr6OF5jjHHb0Hb1eHh4ez7dmMpzXyf7Oxy3/bQ1buWviawC2ohIvIiEAmOA+YWu+RAYCiAisTibsXa67o0RkfzqxWXAZowxphKZNLgl1/VozNOLtvH5plR/h+MWb+1qCD5OIq4axGRgIbAFeFtVk0RkqoiMdF22EDgmIpuBxcCDqnpMVXNxNmV9JSIbcTaNzfJlvMYYU1Yiwj+u60L3prW5/631JB2o/CsAe2tXQ6iAeSKqukBV26pqK1V9zHXuL6o63/WzqurvVLWjqnZR1XkF7l2kql1d5ye4RngZY0ylEh4SxMzxvagdEcKk1xM5fDrT3yGVyFu7GoLNWDfGGK+oFxXOrJsSOJ6exR1zVpPpajKqjLy1qyFYEjHGGK/p3DiaZ0Z3Z+3ekzz8/sZKO2LrfMe6JRFjjKlchndpyO+vaMsHa/fzxw82kpaR7e+QLuLNmohPlz0xxpjqaPJlrTmVmc0r3+9i0eZDPDy8A9f1bFzurWi9JX9yZEQgLMBojDHVjYjwp6s7Mn/yIJrWieD376xn9EvLK80SKRnZ1pxljDGVXufG0bx35wD+dX1Xfkw9zV1vrCE71//ryFrHujHGBAiHQxjduyn/uqEr6/ed5OlF2/wd0vkkEggz1o0xxuDscB/bpykvfrvD75tbZWTlEB7iwOEofx+NJRFjjKkgf76mIy1jI/ntW+s4ftZ/c6ede4l4Z1yVJRFjjKkgEaHBTBvbg5Pp2Tz07nry8vwzjyQjK9crTVlgScQYYypUp0bR/GF4e77ccpgrnvmWuT/srfDZ7d7a1RAsiRhjTIW7ZWALnh3TnfCQIP74wUYGPvE1z321ndwKqpmke2l/dbDJhsYYU+FEhFHdGzOyWyNW7DzOy9/t5N+LtrH/ZAaPX9fF55MSM7JyvDJHBCyJGGOM34gI/VvVpX+rujy1cCvTFydTOyKUKcPb+7Tc9Kxc6keFe+VZlkSMMaYS+P2VbTmRnsWL3+4gJiKEO4a08llZGVm5VhMxxpiqRESYOqozaRnZPP7Zj9SOCOFXvZv5pKz0rFwivDQ6y5KIMcZUEkEO4enR3TmdmcMf3tvIziNneeCqdoQEeXcMVHpWjo3OMsaYqig02MHMm3oxvl9zXlqyk3GzVnAwzbs7JWZk51LDJhsaY0zVFBYcxN+v7cyzY7qTdOAUV0/7jqVeWiolOzeP7Fy1mogxxlR1o7o3Zv7kgdSJDGXCqyv5fNPBcj/Tmyv4giURY4yp1FrXq8W7vxlAp0bR3D13DZ9sOFCu53lza1ywJGKMMZVedI0Q5tzah57NanPvm2v5cO1+j591flfDQEkiIjJMRLaKSLKITCnmmtEisllEkkRkbqH3okRkv4hM93WsxhhTWdUKD+G1iX3oE1+H+99ex4vf7uBcTtnX3PppL5EA6FgXkSBgBjAc6AiMFZGOha5pAzwMDFTVTsBvCz3m78C3vozTGGMCQWRYMK9O6MPl7evzxGc/ctlT3/Le6pQyrbmVvzVuoNRE+gDJqrpTVbOAecCoQtdMAmao6gkAVT2c/4aI9ALqA1/4OE5jjAkINUKDmHVTL+bc2oeYyBB+/856hj+7hNV7jrt1f6B1rDcG9hU4TnGdK6gt0FZElorIChEZBiAiDuDfwIMlFSAit4tIoogkHjlyxIuhG2NM5SQiDG4Tx/y7BzFjXE8ysnMZO+sHPtuYWuq9Ga4+kUDpWC9qKcrC9a5goA1wKTAWeFlEagN3AQtUdR8lUNWZqpqgqglxcXFeCNkYYwKDwyFc3bUhH909iM6Norhr7hpmf7+rxHt+qol4p0/E18uepABNCxw3AQqPT0sBVqhqNrBLRLbiTCr9gcEichdQEwgVkTOqWmTnvDHGVFd1IkOZO6kf981by9RPNnPgZAZ/HNGhyD3UA605axXQRkTiRSQUGAPML3TNh8BQABGJxdm8tVNVf62qzVS1BfAA8F9LIMYYU7TwkCCe/3Uvbu7fnJe/38WYWSvYc+zsBdes3nOCl7/bSY2QIKLCQ7xSrk+TiKrmAJOBhcAW4G1VTRKRqSIy0nXZQuCYiGwGFgMPquoxX8ZljDFVUZBD+OvITjx5Q1e2pJ5i2H++47WluziXk8tTC7fyyxeXkZ2rvDaxt9f6RETVPxvF+0JCQoImJib6OwxjjPG71LQMHn5/I99sPUKt8GBOZ+bwy15N+MvPO1KrUC1ERFaraoIn5dhS8MYYUwU1jK7BqxN6887qFN5YsYe7h7bmyk4NvF6OJRFjjKmiRITRCU0ZndC09Is9ZGtnGWOM8ZglEWOMMR6zJGKMMcZjlkSMMcZ4zJKIMcYYj1kSMcYY4zFLIsYYYzxmScQYY4zHqtSyJyJyGtjq4e3RQJoH7xd1vuC5wu/nHxd1TSxwtExRlxybO9cUF19Rx0X9XPCcr+L39LsvfFxS/J7GXlJ8pb1flt+d0n4O5N+dyvjdFz6uav/dFvw5FohUVc/20lDVKvMCEstx70xP3i/qfMFzhd/PPy7qGk/jLy32ssRf0nExMRc855P4Pf3uyxJ/Zf/dceNzBOzvTmX87kuLOdC/e299/6pqzVkFfOzh+0Wd/7iE9z9245qycud+d+Mv6bion8sbuzvP8PS7L3wcaPGX9WdP2O9O8edKizmQv3t3yy9VVWvOSlQPV6KsDCx+/wnk2CGw4w/k2MHir2o1kZn+DqCcLH7/CeTYIbDjD+TYoZrHX6VqIsYYYypWVauJGGOMqUCWRIwxxnjMkogxxhiPVZskIiKDReRFEXlZRJb5O56yEhGHiDwmIs+JyM3+jqcsRORSEfnO9f1f6u94PCEikSKyWkSu8XcsZSUiHVzf/bsi8ht/x1MWInKtiMwSkY9E5Ep/x1NWItJSRF4RkXf9HYs7XL/nr7u+81+7c09AJBERmS0ih0VkU6Hzw0Rkq4gki8iUkp6hqt+p6p3AJ8Drvoy3MG/ED4wCGgPZQIqvYi3MS7ErcAYIpwJjB6/FD/AH4G3fRFk8L/3ub3H97o8GKmwoqpdi/1BVJwETgF/5MNyLeCn+nap6q28jLVkZP8d1wLuu73ykWwWUZ6ZiRb2AS4CewKYC54KAHUBLIBRYD3QEuuBMFAVf9Qrc9zYQFWjxA1OAO1z3vhtgsTtc99UH3gjA7/5nwBicf8iuCbT4XfeMBJYB4wItdtd9/wZ6BuJ377qvwv6bLefneBjo7rpmrjvPDyYAqOoSEWlR6HQfIFlVdwKIyDxglKo+DhTZ5CAizYA0VT3lw3Av4o34RSQFyHId5vou2gt567t3OQGE+SLO4njpux8KROL8jyxDRBaoap5PA3fx1vevqvOB+SLyKTDXdxFfUKY3vnsBngA+U9U1vo34Ql7+3febsnwOnC0FTYB1uNlSFRBJpBiNgX0FjlOAvqXccyvwqs8iKpuyxv8+8JyIDAaW+DIwN5QpdhG5DrgKqA1M921obilT/Kr6JwARmQAcragEUoKyfv+X4mymCAMW+DSy0pX19/4enDXBaBFpraov+jI4N5T1u68LPAb0EJGHXcmmMijuc0wDpovI1bi5LEogJxEp4lyJMydV9VEfxeKJMsWvquk4k2BlUNbY38eZBCuLMv/uAKjqa94PxSNl/f6/Ab7xVTBlVNbYp+H8w1ZZlDX+Y8CdvgvHY0V+DlU9C0wsy4MComO9GClA0wLHTYADforFE4EcfyDHDha/PwVy7BD48efz2ucI5CSyCmgjIvEiEoqz43O+n2Mqi0COP5BjB4vfnwI5dgj8+PN573P4a8RAGUcXvAmk8tPw1ltd50cA23COMviTv+OsivEHcuwWv8VeneOvqM9hCzAaY4zxWCA3ZxljjPEzSyLGGGM8ZknEGGOMxyyJGGOM8ZglEWOMMR6zJGKMMcZjlkSM8SEROePvGIzxJUsixlQwEQnydwzGeIslEWMqgDh3d1wsInOBjf6OxxhvCeRVfI0JNH2Azqq6y9+BGOMtVhMxpuKstARiqhpLIsZUnLP+DsAYb7MkYowxxmOWRIwxxnjMloI3xhjjMauJGGOM8ZglEWOMMR6zJGKMMcZjlkSMMcZ4zJKIMcYYj1kSMcYY4zFLIsYYYzxmScQYY4zH/h8MtB7OiCp58wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size = 256\n",
    "lr_finder = model.lr_finder(x_train, y_train, batch_size, tolerance=3)\n",
    "_ = lr_finder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07390722033525823"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_finder.get_best_lr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often, this learning rate is a little high, so we instead set it manually to 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.optimizer.set_lr(0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We include the `EarlyStopping` callback to stop training when the validation loss stops improving. After training, this callback will also load the best performing model in terms of validation loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\t[0s / 0s],\t\ttrain_loss: 0.6729,\tval_loss: 0.5150\n",
      "1:\t[0s / 0s],\t\ttrain_loss: 0.5835,\tval_loss: 0.5140\n",
      "2:\t[0s / 0s],\t\ttrain_loss: 0.5536,\tval_loss: 0.5021\n",
      "3:\t[0s / 0s],\t\ttrain_loss: 0.5422,\tval_loss: 0.5063\n",
      "4:\t[0s / 0s],\t\ttrain_loss: 0.5292,\tval_loss: 0.5033\n",
      "5:\t[0s / 0s],\t\ttrain_loss: 0.5207,\tval_loss: 0.5011\n",
      "6:\t[0s / 0s],\t\ttrain_loss: 0.5101,\tval_loss: 0.5014\n",
      "7:\t[0s / 0s],\t\ttrain_loss: 0.5130,\tval_loss: 0.5037\n",
      "8:\t[0s / 0s],\t\ttrain_loss: 0.5085,\tval_loss: 0.4986\n",
      "9:\t[0s / 0s],\t\ttrain_loss: 0.5069,\tval_loss: 0.4990\n",
      "10:\t[0s / 0s],\t\ttrain_loss: 0.4977,\tval_loss: 0.4984\n",
      "11:\t[0s / 0s],\t\ttrain_loss: 0.4908,\tval_loss: 0.5004\n",
      "12:\t[0s / 0s],\t\ttrain_loss: 0.4952,\tval_loss: 0.4988\n",
      "13:\t[0s / 0s],\t\ttrain_loss: 0.4914,\tval_loss: 0.5005\n",
      "14:\t[0s / 0s],\t\ttrain_loss: 0.4887,\tval_loss: 0.5092\n",
      "15:\t[0s / 0s],\t\ttrain_loss: 0.4846,\tval_loss: 0.5100\n",
      "16:\t[0s / 0s],\t\ttrain_loss: 0.4721,\tval_loss: 0.5068\n",
      "17:\t[0s / 0s],\t\ttrain_loss: 0.4830,\tval_loss: 0.5061\n",
      "18:\t[0s / 0s],\t\ttrain_loss: 0.4831,\tval_loss: 0.5063\n",
      "19:\t[0s / 0s],\t\ttrain_loss: 0.4772,\tval_loss: 0.5074\n",
      "20:\t[0s / 1s],\t\ttrain_loss: 0.4773,\tval_loss: 0.5081\n"
     ]
    }
   ],
   "source": [
    "epochs = 100\n",
    "callbacks = [tt.callbacks.EarlyStopping()]\n",
    "log = model.fit(x_train, y_train, batch_size, epochs, callbacks, val_data=val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAD4CAYAAAAdIcpQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXwV9bn48c+TPZAEskIWIGEJm2yyibi3Cm5AFRFtq9harvVS7aIV2mqt1Vvb3lutLdWfWpdaq1S8KNSFuuC1oAjBsgUEAgQSAySEkIXsyfP7YyZwCAk5kJOcLM/79ZrXmfnOd+Y8ZzicJ9/vfGdGVBVjjDHmdAL8HYAxxpiOz5KFMcaYFlmyMMYY0yJLFsYYY1pkycIYY0yLgvwdwJmIi4vT1NRUf4dhjDGdyoYNGw6ranxr9tGpkkVqaioZGRn+DsMYYzoVEdnX2n1YN5QxxpgWWbIwxhjTIksWxhhjWuTVOQsRmQ78HggEnlXVR5uoMwd4EFBgk6reLCKXAo95VBsGzFXVN0TkBeBioNhdN09VN57tBzHGdE41NTXk5uZSWVnp71A6vbCwMFJSUggODvb5vltMFiISCCwGLgdygfUislxVt3nUGQIsAqaqapGIJACo6ipgrFsnBsgC/umx+3tVdamvPowxpvPJzc0lMjKS1NRURMTf4XRaqkphYSG5ubmkpaX5fP/edENNArJUdY+qVgOvAjMb1fkOsFhViwBUNb+J/cwG3lHV8tYEbIzpWiorK4mNjbVE0UoiQmxsbJu10LxJFslAjsdyrlvmKR1IF5E1IrLW7bZqbC7wSqOyR0Rks4g8JiKhTb25iMwXkQwRySgoKPAiXGNMZ2OJwjfa8jh6kyyaevfG9zUPAoYAlwA3Ac+KSO/jOxBJBEYBKz22WYRzDmMiEAPc19Sbq+rTqjpBVSeERUZ7Ea4xxhhf8yZZ5AL9PJZTgLwm6rypqjWquhfYgZM8GswBlqlqTUOBqh5QRxXwPE5312kdraj2IlxjjDG+5k2yWA8MEZE0EQnB6U5a3qjOG8ClACISh9Mttcdj/U006oJyWxuI026aBWxtKZCK6jovwjXGGO8dPXqUP/3pT2e83VVXXcXRo0fPeLt58+axdGnnG9fTYrJQ1VpgAU4X0nbg76qaKSIPicgMt9pKoFBEtgGrcEY5FQKISCpOy+T/Gu36ZRHZAmwB4oCHW4qlqraeyhpLGMYY32kuWdTVnf635u2336Z3796nrdOVeHWdhaq+DbzdqOwBj3kFfuhOjbfN5tQT4qjqZWcYKwA7DpYypl/3+Qcypjv5xYpMtuWV+HSfI5Ki+Pm1I5tdv3DhQnbv3s3YsWMJDg4mIiKCxMRENm7cyLZt25g1axY5OTlUVlZy9913M3/+fODEverKysq48sorueCCC/jkk09ITk7mzTffJDw8vMXYPvjgA+655x5qa2uZOHEiTz75JKGhoSxcuJDly5cTFBTEFVdcwX//93/z2muv8Ytf/ILAwEB69erFxx9/7LNj5I1OdSNBgMy8EksWxhifefTRR9m6dSsbN27ko48+4uqrr2br1q3Hr1V47rnniImJoaKigokTJ3L99dcTGxt70j527drFK6+8wjPPPMOcOXN4/fXX+cY3vnHa962srGTevHl88MEHpKenc8stt/Dkk09yyy23sGzZMr744gtE5HhX10MPPcTKlStJTk4+q+6v1upUySJAhMy84pYrGmM6pdO1ANrLpEmTTrqo7YknnmDZsmUA5OTksGvXrlOSRVpaGmPHjgVg/PjxZGdnt/g+O3bsIC0tjfT0dABuvfVWFi9ezIIFCwgLC+P222/n6quv5pprrgFg6tSpzJs3jzlz5nDdddf54qOekU51b6jw4EC2HfBtE9UYYzz17Nnz+PxHH33E+++/z6effsqmTZsYN25ckxe9hYaeuEwsMDCQ2traFt/H6b0/VVBQEOvWreP666/njTfeYPp057K1p556iocffpicnBzGjh1LYWHhmX60VulULYuwkAC+OFBKXb0SGGAX8RhjWi8yMpLS0tIm1xUXFxMdHU2PHj344osvWLt2rc/ed9iwYWRnZ5OVlcXgwYN56aWXuPjiiykrK6O8vJyrrrqK8847j8GDBwOwe/duJk+ezOTJk1mxYgU5OTmntHDaUqdKFuHBgVTU1LH3cBmDEyL9HY4xpguIjY1l6tSpnHPOOYSHh9OnT5/j66ZPn85TTz3F6NGjGTp0KOedd57P3jcsLIznn3+eG2644fgJ7jvuuIMjR44wc+ZMKisrUVUee8y5F+u9997Lrl27UFW+8pWvMGbMGJ/F4g1prinUEZ0zZpyWXfkwv587lpljTxlgZYzphLZv387w4cP9HUaX0dTxFJENqjqhNfvtVOcsQoMDCQkM8PnQOmOMMafXqbqhBEjvG0GmJQtjTAf3n//5n6xZs+aksrvvvpvbbrvNTxG1TqdKFgAjE3vx3vZDqKrdqdIY02EtXrzY3yH4VKfqhgIYmRzFkWPVHCyxp2oZY0x76XTJYkRiFACZX1pXlDHGtJdOlyyGJ0Yhgp23MMaYdtTpkkXP0CDSYnuy7YDd9sMYY9pLp0sWAMOToqxlYYzxi4iIiGbXZWdnc84557RjNO2nUyaLkUlR5BZVUFxe03JlY4wxrdbphs4CjEzqBcC2AyVMGdR+90YxxrSxdxbCwS2+3WffUXDlo82uvu+++xgwYAB33nknAA8++CAiwscff0xRURE1NTU8/PDDzJw584zetrKyku9+97tkZGQQFBTE7373Oy699FIyMzO57bbbqK6upr6+ntdff52kpCTmzJlDbm4udXV13H///dx4442t+ti+5lXLQkSmi8gOEckSkYXN1JkjIttEJFNE/uZRXiciG91puUd5moh8JiK7RGSJ+8hWrxwfEWW3KzfGtNLcuXNZsmTJ8eW///3v3HbbbSxbtozPP/+cVatW8aMf/ajZu8Q2p+E6iy1btvDKK69w6623UllZyVNPPcXdd9/Nxo0bycjIICUlhXfffZekpCQ2bdrE1q1bj99ptiNpsWUhIoHAYuByIBdYLyLLVXWbR50hwCJgqqoWiUiCxy4qVHVsE7v+NfCYqr4qIk8B3wae9Cbo+MhQEiJD7bYfxnQ1p2kBtJVx48aRn59PXl4eBQUFREdHk5iYyA9+8AM+/vhjAgIC+PLLLzl06BB9+/b1er+rV6/me9/7HuDcYXbAgAHs3LmTKVOm8Mgjj5Cbm8t1113HkCFDGDVqFPfccw/33Xcf11xzDRdeeGFbfdyz5k3LYhKQpap7VLUaeBVo3B77DrBYVYsAVDX/dDsU59Lry4CGp5a/CMw6k8BHJkXZsy2MMT4xe/Zsli5dypIlS5g7dy4vv/wyBQUFbNiwgY0bN9KnT58mn2NxOs21RG6++WaWL19OeHg406ZN48MPPyQ9PZ0NGzYwatQoFi1axEMPPeSLj+VT3iSLZCDHYzmXU5+pnQ6ki8gaEVkrIp5tqDARyXDLGxJCLHBUVRueENLUPgEQkfnu9hkFBQXHy0cm9WJXfhmVNad/qLoxxrRk7ty5vPrqqyxdupTZs2dTXFxMQkICwcHBrFq1in379p3xPi+66CJefvllAHbu3Mn+/fsZOnQoe/bsYeDAgdx1113MmDGDzZs3k5eXR48ePfjGN77BPffcw+eff+7rj9hq3pzgbuoGTI1TZhAwBLgESAH+JSLnqOpRoL+q5onIQOBDEdkCNNUkaDINq+rTwNMAEyZMOF5nRFIUdfXKzkOljE6xZ3IbY87eyJEjKS0tJTk5mcTERL7+9a9z7bXXMmHCBMaOHcuwYcPOeJ933nknd9xxB6NGjSIoKIgXXniB0NBQlixZwl//+leCg4Pp27cvDzzwAOvXr+fee+8lICCA4OBgnnzSqx75duVNssgF+nkspwB5TdRZq6o1wF4R2YGTPNarah6Aqu4RkY+AccDrQG8RCXJbF03t87RGJjWc5C6xZGGMabUtW06MwoqLi+PTTz9tsl5ZWVmz+0hNTWXr1q2A83CjF1544ZQ6ixYtYtGiRSeVTZs2jWnTpp1F1O3Hm26o9cAQd/RSCDAXWN6ozhvApQAiEofTLbVHRKJFJNSjfCqwTZ3OvFXAbHf7W4E3zyTwftE9iAwNspPcxhjTDlpsWahqrYgsAFYCgcBzqpopIg8BGaq63F13hYhsA+qAe1W1UETOB/6fiNTjJKZHPUZR3Qe8KiIPA/8G/nwmgQcECMMTo2z4rDGm3W3ZsoVvfvObJ5WFhoby2Wef+SmitufVRXmq+jbwdqOyBzzmFfihO3nW+QQY1cw+9+CMtDprI5KiWLI+h7p6JTDAnm1hTGfV2Z5PM2rUKDZu3OjvME7Rlo/J7pS3+2gwMimKipo6sguP+TsUY8xZCgsLo7CwsE1/6LoDVaWwsJCwsLA22X+nvN1Hg4bbfmTmlTAovvmbexljOq6UlBRyc3PxHBpvzk5YWBgpKSltsu9OnSwGJ0QQHChk5hUzY0ySv8MxxpyF4OBg0tLS/B2GaUGn7oYKCQogvU+kjYgyxpg21qmTBbi3/cgrsf5OY4xpQ50+WYxIjKLwWDWHSqr8HYoxxnRZnT5ZjExuOMlt11sYY0xb6fTJYnhiFCLYeQtjjGlDnT5ZRIQGkRrb057JbYwxbajTJwtwzltkHrBuKGOMaStdI1kkRZFzpILiihp/h2KMMV1Sl0gWDbcr325PzjPGmDbRJZLFCI9nWxhjjPG9LpEsEiLDiI8MteGzxhjTRrpEsoATV3IbY4zxPa+ShYhMF5EdIpIlIgubqTNHRLaJSKaI/M0tGysin7plm0XkRo/6L4jIXhHZ6E5jW/NBRiZFkZVfRlVtXWt2Y4wxpgkt3nVWRAKBxcDlOM/aXi8iyz2eeIeIDAEWAVNVtUhEEtxV5cAtqrpLRJKADSKyUlWPuuvvVdWlvvggIxJ7UVuv7DxYxqiUXr7YpTHGGJc3LYtJQJaq7lHVauBVYGajOt8BFqtqEYCq5ruvO1V1lzufB+QD8b4K3lPDiKhtdr2FMcb4nDfJIhnI8VjOdcs8pQPpIrJGRNaKyPTGOxGRSUAIsNuj+BG3e+oxEQlt6s1FZL6IZIhIxukejtI/pgcRoUE2IsoYY9qAN8miqQfjNr4feBAwBLgEuAl4VkR6H9+BSCLwEnCbqta7xYuAYcBEIAa4r6k3V9WnVXWCqk6Ij2++URIQIAxPjLRkYYwxbcCbZJEL9PNYTgHymqjzpqrWqOpeYAdO8kBEooC3gJ+p6tqGDVT1gDqqgOdxurtaZWRSL7YfKKG+3p5tYYwxvuRNslgPDBGRNBEJAeYCyxvVeQO4FEBE4nC6pfa49ZcBf1HV1zw3cFsbiIgAs4Ctrfkg4FycV15dR3bhsdbuyhhjjIcWk4Wq1gILgJXAduDvqpopIg+JyAy32kqgUES2AatwRjkVAnOAi4B5TQyRfVlEtgBbgDjg4dZ+mBGJdiW3Mca0hRaHzgKo6tvA243KHvCYV+CH7uRZ56/AX5vZ52VnGmxL0vtEEhwoZOaVcO2YJF/v3hhjuq0ucwU3QEhQAEMSItlmNxQ0xhif6lLJAhpu+1GM09gxxhjjC10uWYxIiuJwWTX5pVX+DsUYY7qMLpcsRiY5t/qwmwoaY4zvdLlkMTwxEsBuV26MMT7U5ZJFZFgwA2J72PBZY4zxoS6XLMA5yW3JwhhjfKeLJote7D9STklljb9DMcaYLqFLJouGZ3Jvt9aFMcb4RJdMFiPtth/GGONTXTJZJESFERcRaldyG2OMj3TJZAF2ktsYY3ypyyaLEUlR7DpUSlVtnb9DMcaYTq/LJouRSVHU1iu7DpX5OxRjjOn0unCysNt+GGOMr3TZZDEgpgc9QwLtth/GGOMDXiULEZkuIjtEJEtEFjZTZ46IbBORTBH5m0f5rSKyy51u9SgfLyJb3H0+4T5e1WcCAoThiXaS2xhjfKHFZCEigcBi4EpgBHCTiIxoVGcIsAiYqqojge+75THAz4HJwCTg5yIS7W72JDAfGOJO033xgTyNTIpi+4ES6uvt2RbGGNMa3rQsJgFZqrpHVauBV4GZjep8B1isqkUAqprvlk8D3lPVI+6694DpIpIIRKnqp+4jWf8CzPLB5znJyKReHKuuY9+Rcl/v2hhjuhVvkkUykOOxnOuWeUoH0kVkjYisFZHpLWyb7M6fbp8AiMh8EckQkYyCggIvwj2h4bYfdt7CGGNax5tk0dS5hMb9OkE4XUmXADcBz4pI79Ns680+nULVp1V1gqpOiI+P9yLcE4b0iSAoQGxElDHGtJI3ySIX6OexnALkNVHnTVWtUdW9wA6c5NHctrnu/On22WqhQYEM6RNpJ7mNMaaVvEkW64EhIpImIiHAXGB5ozpvAJcCiEgcTrfUHmAlcIWIRLsntq8AVqrqAaBURM5zR0HdArzpk0/UyAgbEWWMMa3WYrJQ1VpgAc4P/3bg76qaKSIPicgMt9pKoFBEtgGrgHtVtVBVjwC/xEk464GH3DKA7wLPAlnAbuAdH36u40YmRXG4rIr80sq22L0xxnQLQd5UUtW3gbcblT3gMa/AD92p8bbPAc81UZ4BnHOG8Z6xkUknbleeMDSsrd/OGGO6pC57BXeD4W6ysJPcxhhz9rp8sogKC6Z/TA8bPmuMMa3Q5ZMFOF1R1rIwxpiz122SRXZhOaWVNf4OxRhjOqVukSwaruTefqDUz5EYY0zn1C2SxYlnW9h5C2OMORvdIlkkRIYSFxFiF+cZY8xZ6hbJQsSebWGMMa3RLZIFOF1Ru/JLqa6t93coxhjT6XSjZBFFTZ2y85Cd5DbGmDPVbZLF+AHRBAYIS9bntFzZGGPMSbpNskjqHc43Jvfnb+v2W+vCGGPOULdJFgDf/2o6PUMCefit7f4OxRhjOpVulSyie4Zw91fT+XhnAat25Le8gTHGGKCbJQuAb543gLS4njz8j23U1NnIKGOM8YZXyUJEpovIDhHJEpGFTayfJyIFIrLRnW53yy/1KNsoIpUiMstd94KI7PVYN9a3H61pIUEB/PSq4ewuOMbfPtvfHm9pjDGdXosPPxKRQGAxcDnOs7PXi8hyVd3WqOoSVV3gWaCqq4Cx7n5icJ6K90+PKveq6tJWxH9WvjI8gamDY3ns/Z3MGptMrx7B7R2CMcZ0Kt60LCYBWaq6R1WrgVeBmWfxXrOBd1S1/Cy29SkR4WdXj6Ckoobff7DL3+EYY0yH502ySAY8L07Idcsau15ENovIUhHp18T6ucArjcoecbd5TERCvQvZN4YnRnHjxP785dNsdheUtedbG2NMp+NNspAmyrTR8gogVVVHA+8DL560A5FEYBSw0qN4ETAMmAjEAPc1+eYi80UkQ0QyCgoKvAjXez+6Ip2w4EB+9bYNpTXGmNPxJlnkAp4thRQgz7OCqhaqapW7+AwwvtE+5gDLVLXGY5sD6qgCnsfp7jqFqj6tqhNUdUJ8fLwX4XovLiKUBZcN5v3t+fxrl28TkTHGdCXeJIv1wBARSROREJzupOWeFdyWQ4MZQOM/1W+iURdUwzYiIsAsYOuZhe4bt01NpX9MDx7+x3ZqbSitMcY0qcVkoaq1wAKcLqTtwN9VNVNEHhKRGW61u0QkU0Q2AXcB8xq2F5FUnJbJ/zXa9csisgXYAsQBD7fuo5yd0KBAFl05jB2HSlmSYfeNMsaYpohq49MPHdeECRM0IyPD5/tVVW58ei2788tYde8lRIXZUFpjTNchIhtUdUJr9tHtruBuiojwwDUjOFJezeIPs/wdjjHGdDiWLFznJPdi9rkpPLdmL/sKj/k7HGOM6VAsWXi4d9pQggMD+NXbX/g7FGOM6VAsWXhIiArjzksG8W7mQT7dXejvcIwxpsOwZNHI7RcOJLl3OA+/tY26+s5z8t8YY9qSJYtGwoIDue/KYWTmlfD6hlx/h2OMMR2CJYsmXDs6kXP79+a3/9xBWVWtv8Mxxhi/s2TRBBHh/mtGUFBaxZMf2VBaY4yxZNGMcf2jmTU2iWf+tZecI36/q7oxxviVJYvT+PH0YQQI/PpdG0prjOneLFmcRlLvcOZfNIh/bD5ARvYRf4djjDF+Y8miBXdcPJA+UaH88h/bqLehtMaYbsqSRQt6hATx42nD2JRbzBsbv/R3OMYY4xeWLLzwtXHJjE7pxW/e3UF5tQ2lNcZ0P5YsvBAQ4AylPVhSyf/7vz3+DscYY9qdJQsvTUyN4erRiTz5f7t5fs1eO39hjOlWvEoWIjJdRHaISJaILGxi/TwRKRCRje50u8e6Oo/y5R7laSLymYjsEpEl7iNbO7SHZozk/EGx/GLFNuY+s5b9hXb9hTGme2gxWYhIILAYuBIYAdwkIiOaqLpEVce607Me5RUe5TM8yn8NPKaqQ4Ai4Ntn/zHaR2xEKM/Pm8hvrh/N9rwSpj3+MX/5NNtaGcaYLs+blsUkIEtV96hqNfAqMLM1byoiAlwGLHWLXgRmtWaf7UVEmDOxHyt/cBET02J44M1Mbn52rV3lbYzp0rxJFslAjsdyrlvW2PUisllElopIP4/yMBHJEJG1ItKQEGKBo6raMLSouX0iIvPd7TMKCgq8CLd9JPUO58XbJvLodaPY+qXTynhp7T5rZRhjuiRvkoU0Udb4F3EFkKqqo4H3cVoKDfq7Dwq/GXhcRAZ5uU+nUPVpVZ2gqhPi4+O9CLf9iAhzJ/Vn5Q8uYvyAaO5/Yyvf+PNn1sowxnQ53iSLXMCzpZAC5HlWUNVCVa1yF58Bxnusy3Nf9wAfAeOAw0BvEQlqbp+dSXLvcP7yrUn819dGsSnnKNMf/5iXP9uHqrUyjDFdgzfJYj0wxB29FALMBZZ7VhCRRI/FGcB2tzxaRELd+ThgKrBNnV/RVcBsd5tbgTdb80H8TUS4ebLTyhjbvzc/XbaVb/55HblF1sowxnR+LSYL97zCAmAlThL4u6pmishDItIwuukuEckUkU3AXcA8t3w4kOGWrwIeVdVt7rr7gB+KSBbOOYw/++pD+VNKdA/++u3JPDzrHP69v4jpj/+LV9btt1aGMaZTk870IzZhwgTNyMjwdxheyzlSzn2vb+aT3YVclB7Po9eNIql3uL/DMsZ0MyKywT13fNbsCu421C/GaWX8cuZIMrKPMO2xj1my3loZxpjOx5JFGwsIEL45JZWV37+IkclR3Pf6Fr79YgZFx6r9HZoxxnjNkkU76RfTg7/dfh4/v3YEq3cd5po/rGZjzlF/h2WMMV6xZNGOAgKE26amsfS7UxCBG576hBc/ybZuKWNMh2fJwg9Gp/Tmre9dyMXp8fx8eSYLXvk3ZVX2nAxjTMdlycJPevUI5ulvTmDhlcN4d+tBZvxhNV8cLPF3WMYY0yRLFn4UECDccfEg/nb7ZMqqapm1eA1LN+T6OyxjjDmFJYsOYPLAWN6660LO7R/NPa9t4r6lm6msqfN3WMYYc5wliw4iPjKUl749me9dNpglGTl87U+fsPfwMX+HZYwxgCWLDiUwQPjRFUN5/raJHCiu4No/rOadLQf8HZYxxliy6IguHZrAW3ddyOCECL778uc8tGIb1bX1/g7LGNONWbLooJJ7h/P3/5jCvPNTeW7NXm58+lPyjlb4OyxjTDdlyaIDCwkK4MEZI1l887nsOlTG1U/8i4925Ps7LGNMN2TJohO4enQiyxdMpU9UGLe9sJ7f/XMHdfb4VmNMO7Jk0UkMjI9g2Z1TmX1uCk98mMXVT/yLd7YcsGd+G2PahSWLTiQ8JJDf3jCGxTefS3VdPd99+XOueuJfvLvVkoYxpm15lSxEZLqI7BCRLBFZ2MT6eSJSICIb3el2t3ysiHzqPkVvs4jc6LHNCyKy12Obsb77WF3b1aMTee8HF/P4jWOprq3njr9+ztV/WM27Ww/aTQmNMW2ixSfliUggsBO4HMjFeSb3TR6PR0VE5gETVHVBo23TAVXVXSKSBGwAhqvqURF5AfiHqi71NtjO9qS89lBbV8+KzXk88UEWew8fY0RiFHd/dQhXjOiDiPg7PGNMB9BeT8qbBGSp6h5VrQZeBWZ6s3NV3amqu9z5PCAfiD/bYM2pggID+Nq4FN77wUX8bs4Yyqtr+Y+XNnDNH1bzz0xraRhjfMObZJEM5Hgs57pljV3vdjUtFZF+jVeKyCQgBNjtUfyIu81jIhLa1JuLyHwRyRCRjIKCAi/C7Z6CAgO47twU3v/hxfz3DWMoq6pl/ksbuPaPq3l/2yFLGsaYVvEmWTTVl9H4l2cFkKqqo4H3gRdP2oFIIvAScJuqNlyKvAgYBkwEYoD7mnpzVX1aVSeo6oT4eGuUtCQoMIDZ41P44IcX89vZoympqOX2v2Qw449r+GC7JQ1jzNnxJlnkAp4thRQgz7OCqhaqapW7+AwwvmGdiEQBbwE/U9W1HtscUEcV8DxOd5fxkaDAAG6Y0I8PfnQxv5k9mqMV1Xz7xQxmLl7Dh19Y0jDGnBlvksV6YIiIpIlICDAXWO5ZwW05NJgBbHfLQ4BlwF9U9bWmthHnLOwsYOvZfgjTvODAAOZM6MeHP7qE31w/miPHqvnWCxnM+tMnbNhX5O/wjDGdRIvJQlVrgQXASpwk8HdVzRSRh0RkhlvtLnd47CbgLmCeWz4HuAiY18QQ2ZdFZAuwBYgDHvbZpzKnCA4MYM7Efqy65xJ+ff0o8ksqmf3UJ/x02RaKK2r8HZ4xpoNrcehsR2JDZ32nrKqWx97byfNr9hLTM5T7rxnOjDFJNtzWmC6ovYbOmi4oIjSI+68ZwfIFF5DcO4y7X93ILc+tY1+hPXDJGHMqSxbd3DnJvfjfO6fyixkj+ff+o1zx2Mf88cNd9vwMY8xJOleyqK+FOutf97XAAOHW81P54EcX89Xhffjvf+7kqif+xWd7Cv0dmjGmg+hc5yySAjVjfgQE94SwXhDe23k9PjVabry+RyyERvr7Y3R4q77I5/43t5JbVMEN41P4yVXDie4Z4u+wjDFnyRfnLDpXshg+QDOeWgCVR92p2JkqPOYrizn1msEGApP/A776IASHt1/gnVBFdR1PfLiLZz7eQ1R4MD+5ajjXnzW3qcQAABnCSURBVJvs0xPgxeU1BAYKEaFBPtunMeZU3S9ZeDMaqr4eqkubTiK562HD8xA/DK57GhLHtE/gndiOg6X8ZNkWNuwr4ryBMTw8axSDEyLOaB/HqmrZlV/GzoOl7DhUyk53OlRSRWCAMCalFxcMjmPq4DjG9Y8mJKhz9Y4a09FZsjgbuz+EN+6EY4fhsp/C+XdBQKBvAuyi6uuVJRk5/Ort7VTU1PHdiwdx56WDCQs++bhV1tSxp+AYOw+5ScFNDrlFJ54dHhYcwJCESNL7RJLeJ4LSylpWZx1mc+5R6hXCgwOZlBZzPHkM6xtJQIAN5zWmNSxZnK3yI/CP78O2N2HAVPjaU9C7f+v328UdLqvikbe2s+zfX5Ia24P5Fw3iUEnl8eSwr7D8+ONegwKEQfERpPeNZGifCNL7RDK0byQp0T0IbOLHv7iihs/2FLIm6zCrsw6zu8AZwhvbM4TzB8dxweBYpg6OIyW6R7t+ZtOOaquhusyZwqPt/KIPWbJoDVXY9Cq8fS+IwFW/hdE3OvPmtFbvOszP3thCdmE5IpAa25P0PhEM7RNJel+n1ZAa27NV3UkHiytZk3X4ePLIL3VuPZYa28NNHnFMGRhrJ947mtJDsP8TqCqF6mNQVXYiAVQfc8vLPNYdc7qNq49BXfXJ+4oZCH1HQ+Jo57XvaIjs45/P1clZsvCFomxYdgfs/xRGfg2u/h30iPHte3RBVbV17Cssp190D8JD2rYbT1XJyi9jtZs81u45QllVLSIwMimKIQmRJESGEh8ZSkJUGAmRoc4UFWYnz9tLWT6sfhwy/gy1lSevCwyBkJ4QEgmhEe68+xoa6bEccWJ96UE4uBkObIaj+07sK6KPmzhGnUgi0WkQYOe5qK9zkm5NBdQcg+pyqHEmGXSpJQufqK+DNb+HVY9AzwSY9ScYdKnv38f4RE1dPZtzi1mTdZhPdh8m50gFBaVVVNedeiFhj5BAN3mEER8VenzeSSbOfN9eYfQKD/bDJ+kCjh2GNY/DumehrgrG3AQTb4eecScSQFArW38VR+HQVidxHNwMB7dAwRfOdVfgJKG+53i0QkZB/PDWv29bU3V+3D0H4VSVnLxcWey2xMqdBFBTcWK+uvzkxFBX1exbyS9KLFn4VN5G+N/vwOGdcN6d8JWfQ3CY7/Z/dD9kr3H+cRNGOFN4b9/tvxtTVYorasgvrSK/pIqCskryS6qc5dIq8ksqKXDny6pqT9pWBL42LpkfXp5u50S8dawQPnkC1j0DtRUwag5c/GOIHdQ+719TCQXb3QSyxU0iW53/WwABQRAa5bRSgsMhuIczhfRoej44/OS6IT0hKNQZXVlf28JUd2K+rubk5dpKqCw5eaj/8YRQAlp3+s8ZFOYm3R7O9WXH4zzNZ2mirqRdYMnC56rL4f2fw7qnnb9Orn/G+UvlTKk6zefs1U6CyF4NxftPrReVDH1GOomj4TUuveP/VdSJlVfXeiSSSv69/ygvrd0HCrdMGcB/XjrYzoU0p/wIfLoYPnvK+at41Gy4+D6IG+LvyJwf6SN74eAmOJTp/CDXVLhdM+Ue8xWN/kovp/lrs1ohINhJOCddONzLSWInlTVaDvUoD2ryAaJnzM5ZtKVd78Obd0JFEVz2M5iy4PRDbFWd8x/Zq51p3xoodp9G2yPWGXWVeoEzhfWCQ9sgP9N93QYFO6DevZVJQJCTMBJGQJ8RkDDSSSS9UuwEfBvJO1rB4+/vZOmGXHqGBHHHJYP41tS0Nj8f02lUHIW1f4K1Tzp/GY/8Gly8EBKG+Tuy1lN1WgAeffzOVOn8XwwIdF/dKTDo5OWm6nSw4fiWLNrasUJYcRd88Q8YcIE7xNZ9aKAqHNlzIjFkr4aSL511PeJOJIbUCyBuaMsn4OpqoDDL+YvoUKaTQA5tO7k1EtoLEoY7CSR2sDPct/cA57Ujd2dVljiJtGiv85dfw3xRtnOCc8g0SJ8GyeP9/p9s56FSfvPuDt7ffog+UaF8/6vp3DA+haDAbnoCtbIY1j7ltCaqimH4DLhkofPHi+k02i1ZiMh04PdAIPCsqj7aaP084LeA+2vJH1X1WXfdrcDP3PKHVfVFt3w88AIQDrwN3K0tBOOX51mowsaX4Z37QAJhyp3Oj3r2aig94NTpGX8iMQy4AOKH+q4FUFkC+dudE3wNCSTfbWJ7Cu0F0R7J4/hrf4ge0LZj1uvroezgyYnAc7680Q0Jw2MgJg2iU51RL/vXOn23PWJhyBVO4hh0mdMC85P12Ud49J0v2LCviIHxPfnxtGFMG9mn+zzvo6rU6Wr65I9Of/vQq50kkTja35GZs9AuyUJEAoGdwOU4z+NeD9ykqts86swDJqjqgkbbxgAZwAScTsENwHhVLRKRdcDdwFqcZPGEqr5zulj8+vCjI3th2X9AzmfO8L3UC9yupQud/tr2/BFRdbrHju53p33Oa9G+E8s15SdvEx59chLplQIS0OgkXd3JJ+60iTLPE35VpU5COLrv5OGSEgC9+jnJICbNGdp4fD711CRQUQRZH8DOlZD1nrMcEAT9p0D6dGeKG9y2x7QJqsp72w7xm5U7yMovY1z/3iycPozJA2PbPZZ2U1XmnK/75A9QccQ59pcshKRx/o7MtEJ7JYspwIOqOs1dXgSgqr/yqDOPppPFTcAlqvof7vL/Az5yp1WqOqypes3x+5Py6uuc8eSRfTv2uQNV56/5o/s8EohHUjm6/9Sx8J4ksFEfbKBb5tk/G+iMuIgecGpS6N0fAs9yKGpdLXyZATvfdZJHvvs3Scwgp8WRPg36n9+uAwBq6+p5/fNcHntvFwdLKvnKsAR+PH0YQ/ueZWutqhSKc90p58S8KkQkOH+MRPQ5eT482jfXEtRUOi3i0oMer3nu60GnBVtRBIMvh0sWQcr41r+n8TtfJAtvrlhKBnI8lnOByU3Uu15ELsJphfxAVXOa2TbZnXKbKD+FiMwH5gP07+/nW3IEBEJUon9j8IaIM869Z5xzHqAxVWdUi8iJpHA8GQT6NxEGBkH/85zpqw86iW3nSmda/2fnJGtIJAy+zDnXMeQKiIh3Wjx11c4ggboaj2GMNU4COl7uvnqu0zrnwrHgcGeoYqPXoOBwbpzQj5ljk3nhk2z+tCqL6b//mOvGpfDDK9JJ7u1xB+P6OudHt3EiOL6cc2oXogRCVJLTIis71HQiDwhyrgE6nkCaSCo945yTzw2JoMQzKbhTRVETxzzU+QMoMtE5nhNvh36TfPmvaroAb5JFU78cjZsjK4BXVLVKRO4AXgQuO8223uzTKVR9GnganJaFF/GalohAz07SldK7P0z6jjNVH4O9H59odWx706kjAaBt/WQ/ISw4nDuCwpgfGcbR0EDyM4WCzBBqoqJIjgomuOyAM8ih8dj5sN5Ot1yvfk7XWq8Ud+rnvEb2PXFiX9VpeZTlO4mj7JDHvPtaegAObIJjBacfpy8BThKJTHRaff2nOH/sRCaeSA6RiU6rpSO3lE2H4E2yyAX6eSynAHmeFVTV8wzmM8CvPba9pNG2H7nlKafbpzGnCOkJQ690JlXnQqzdHzpJJCDYHdIY7HSBBQS5r+6y53zjdRLgtEpqKpy/6lt4DaipJKa2gvDyY+w/VEh+cSn5JQEERw8necwMElIGnUgEvVLObHCBiDvuPqrl8zT19c55heNJpcB5r4aE0DPe76PLTNfhTbJYDwwRkTSc0U5zgZs9K4hIoqq6Q4OYAWx351cC/yUi0e7yFcAiVT0iIqUich7wGXAL8IfWfRTTrYg4zyPx4zNJwoGhOMNtn/l4Dys251F5sJ4x/Xrz9cn9uXZAUttepxEQcKK70Yaymjbm7dDZq4DHcYbOPqeqj4jIQ0CGqi4XkV/hJIla4AjwXVX9wt32W8BP3F09oqrPu+UTODF09h3gex1y6KwxXiour+F//53Ly5/tJyu/jMiwIK4/N4WvT+7PkD5tN3S5vLqW9dlFbNhXRJ+oUMYPiGZIQmSTt4I33ZNdlGdMB6SqrNt7hJc/28+7Ww9SXVfPpNQYvn5ef6af05fQoNa1Nipr6vj3/qN8uqeQT3cfZmPOUWrqTv5/HBEaxNh+vTl3QDTn9u/NuP7RdrPEbsyShTEdXGFZFUs35PK3dfvZV1hOTM8Qbhifwk2T+pMa19OrfTTcZXftnkI+2X2YjOwiqmrrCRAYldyLKYPimDIolgkDoikoreLz/UV8vr+IDfuOsuNgCe7zqBiSEMG5/aM5d0Bvxg+IZmBchD2FsJuwZGFMJ1Ffr6zZfZiX1+7nve2HqKtXLhwSx9cn9+crw/sQ7HE7kbp6ZfuBEj7d7SSHdXuPcKzaGfU0rG8k5w+K4/xBsUxMi2mxtVBWVcvmnKNu8iji3zlHOVru3IMsKiyIcf2jGT8gmnP7RzOmXy8iw6z10RVZsjCmEzpUUsmS9Tm8sm4/B4orSYgM5caJ/YiLCOWT3c7DnYornB/0gfE9OX9QLFMGxnHewBhiI1p3F1JVZc/hY07i2F/E5/uOsjO/FFVnzEB6QiRD+kQwKD6CQQkRDI6PIC2up91QsZOzZGFMJ1ZbV89HOwp4+bN9fLSzAFVI7h3O1MGxTHETRN9ePnyeSjNKKmvYuN9pfWzOLSYrv4yconIafhpEnLgGxTckkZ7H5+MiQrrP/bI6MUsWxnQRB4orqK1T+sV0jIcvVdbUkV14jN35x9hdUMbugjKy8svYU3CMipoTFwL2Cg9mUHzP4y2RQfERDOsbSUp0uCWRDqS9bvdhjGljib3CW67UjsKCAxnWN4phfaNOKq+vVw6UVLI7v+x4Etmdf4yPdhbw2oYTd/DpGxXGpLQYJqXFMDkthsEJEZY8OjlLFsYYrwUECMm9w0nuHc5F6fEnrSuuqGFPQRlbvyxmXXYRa/cUsnyTc2OGmJ4hTEyNZlJaLJPTYhieGGXXgXQy1g1ljGkTqsr+I+V8tvcI69xp/xHn1vkRoUFMSI1mYqrT8hiV0qvV15+Y5lk3lDGmwxIRBsT2ZEBsT+ZMcG4vd6C44njiWLf3CB/t2AFAaFAA4/r3Pt7ymJAabcmjg7GWhTHGbwrLqlifXcS6vUdYn32EzLxi6hUGxvXkDzePY2SS/56W2JXYaChjTJdSWlnD6l2HeXBFJkXHavjp1cO5ZcqAdj85nnOknNc25FJVWwcK9arUu696fLlh3ulya1xHVYmLCOWq0YmM69fbryf4LVkYY7qkI8equee1TXz4RT6Xj+jDb2ePpnePtn86Ym1dPc+vyeZ37+2ksraO4MAAAgQCRAgQcZ4X5vEaIE53W3N1DhRXUl1bT7+YcK4dncSMsUmnjDBrD5YsjDFdlqry3JpsHn1nO/ERofz+pnFMTI1ps/fb+mUxi/53C1u+LOarwxP45axzWj2kuaSyhn9mHmL5pjzWZB2mrl5J7xPBjDFJXDsmiQGx3t0frLUsWRhjurwtucUseOVzco6U84OvpnPnpYN9Ouy2orqOxz/YybP/2kt0jxB+MWMkV43q6/Nuo8KyKt7ecoAVmw6wLvsIAGP69WbGmCSuGZ1In6i2u1rfkoUxplsorazhZ29s5c2NeUwZGMvjc8f65Md1TdZhfrJsC/sKy7lxQj9+ctVwevVo+5sp5h2t4B+b81i+KY+tX5YgApPTYpgxJpkrz+lLdE/fdrlZsjDGdBuqytINuTzwZibhIYH8z5wxXDo04az2dbS8mkfe2s5rG3JJi+vJf31tFFMG+ee59LsLylixyUkcewqOERQgXJQez4wxSVw+og89Q1t/hUO7JQsRmQ78HudJec+q6qPN1JsNvAZMVNUMEfk6cK9HldHAuaq6UUQ+AhKBCnfdFaqaf7o4LFkYY7Lyy1jwt8/54mAp37kwjXunDSMkKKDlDXESzorNB3hoRSZHy2uYf9FA7vrKEMKC/X9Nh6qSmVfCik15rNiUR15xJWHBAfSL7uGMsILjo6zqFRSlvt7Z1nOUlrr78lze9PNpbZ8sRCQQ2AlcDuTiPJP7JlXd1qheJPAWEAIsUNWMRutHAW+q6kB3+SPgnsb1TseShTEGnBsdPvLWdl5au48xKb34w03n0j/29Ddh/PJoBfe/sZUPv8hnTEovfnXdaEYktf/IJG/U1ysb9hfx1uYD5JdWIiIIJ0ZZNczTMPLKc93xEVkgOPO/nDWqXa7gngRkqeoeABF5FZgJbGtU75fAb4B7mtnPTcArZxmnMcYcFxYcyC9nncPUwbH8eOlmrn7iX/zXdaO4dkzSKXXr6pWXPs3mtyt3UK9w/zUjmHd+aoe+N1VAgDAxNcZno79+6YN9eNN2SwZyPJZz3bLjRGQc0E9V/3Ga/dzIqcnieRHZKCL3SzNDD0RkvohkiEhGQUGBF+EaY7qL6eck8vbdF5LeN5LvvfJvFr6+mYrqE7dQ33GwlNlPfcKDK7YxPjWGf/7gIr59QVqHThQdlTcti6aO6vG+KxEJAB4D5jW7A5HJQLmqbvUo/rqqful2X70OfBP4yylvpPo08DQ43VBexGuM6UZSonuwZP55PPb+Tv700W427Cvif+aM4f1th3jy/3YTGRbM4zeOZebYJLtNeit4kyxygX4eyylAnsdyJHAO8JH7D9EXWC4iMzzOR8ylUatCVb90X0tF5G843V2nJAtjjGlJUGAA904bxpSBcXx/yUZm/HENANeNS+Zn14wgxsdDUbsjb5LFemCIiKQBX+L88N/csFJVi4G4huXGJ67dlscNwEUedYKA3qp6WESCgWuA91v9aYwx3doFQ+J45+4LWbwqi8uGJZzyzA1z9lpMFqpaKyILgJU4Q2efU9VMEXkIyFDV5S3s4iIgt+EEuSsUWOkmikCcRPHMWX0CY4zxEB8ZyoMzRvo7jC7HLsozxpguzhcX5Xl3JYsxxphuzZKFMcaYFlmyMMYY0yJLFsYYY1pkycIYY0yLLFkYY4xpkSULY4wxLepU11mISCmww99xeCEOOOzvILzQGeLsDDGCxelrFqdvDVXVyNbsoPWPYGpfO1p7YUl7EJEMi9M3OkOMYHH6msXpWyLS6quZrRvKGGNMiyxZGGOMaVFnSxZP+zsAL1mcvtMZYgSL09csTt9qdZyd6gS3McYY/+hsLQtjjDF+YMnCGGNMizpkshCR6SKyQ0SyRGRhE+tDRWSJu/4zEUlt5/j6icgqEdkuIpkicncTdS4RkWIR2ehOD7RnjB5xZIvIFjeGU4bPieMJ91huFpFz/RDjUI/jtFFESkTk+43q+OV4ishzIpIvIls9ymJE5D0R2eW+Rjez7a1unV0icqsf4vytiHzh/rsuE5HezWx72u9IO8T5oIh86fFve1Uz2572d6Ed4lziEWO2iGxsZtt2OZ7N/Q612fdTVTvUhPPkvN3AQCAE2ASMaFTnTuApd34usKSdY0wEznXnI4GdTcR4CfCPDnA8s4G406y/CngHEOA84LMO8O9/EBjQEY4nzpMezwW2epT9Bljozi8Eft3EdjHAHvc12p2Pbuc4rwCC3PlfNxWnN9+RdojzQZxHMbf0vTjt70Jbx9lo/f8AD/jzeDb3O9RW38+O2LKYBGSp6h5VrQZeBWY2qjMTeNGdXwp8RUSkvQJU1QOq+rk7XwpsB5Lb6/19bCbwF3WsBXqLSKIf4/kKsFtV9/kxhuNU9WPgSKNiz+/fi8CsJjadBrynqkdUtQh4D5jennGq6j9VtdZdXAuktNX7e6uZ4+kNb34XfOZ0cbq/NXOAV9rq/b1xmt+hNvl+dsRkkQzkeCzncuoP8fE67n+GYiC2XaJrxO0CGwd81sTqKSKySUTeERF/PRRYgX+KyAYRmd/Eem+Od3uaS/P/CTvC8QToo6oHwPkPCyQ0UaejHddv4bQgm9LSd6Q9LHC7y55rptukIx3PC4FDqrqrmfXtfjwb/Q61yfezIyaLploIjcf3elOnzYlIBPA68H1VLWm0+nOcrpQxwB+AN9o7PtdUVT0XuBL4TxG5qNH6DnEsAUQkBJgBvNbE6o5yPL3VkY7rT4Fa4OVmqrT0HWlrTwKDgLHAAZwunsY6zPEEbuL0rYp2PZ4t/A41u1kTZac9nh0xWeQC/TyWU4C85uqISBDQi7Nr2p41EQnG+Qd6WVX/t/F6VS1R1TJ3/m0gWETi2jNG973z3Nd8YBlOc96TN8e7vVwJfK6qhxqv6CjH03WooavOfc1vok6HOK7uictrgK+r21ndmBffkTalqodUtU5V64Fnmnn/jnI8g4DrgCXN1WnP49nM71CbfD87YrJYDwwRkTT3L825wPJGdZYDDWfvZwMfNvcfoS24fZZ/Brar6u+aqdO34TyKiEzCOdaF7RWj+749RSSyYR7nhOfWRtWWA7eI4zyguKEJ6wfN/sXWEY6nB8/v363Am03UWQlcISLRbrfKFW5ZuxGR6cB9wAxVLW+mjjffkTbV6BzZ15p5f29+F9rDV4EvVDW3qZXteTxP8zvUNt/Ptj5jf5Zn+a/CObO/G/ipW/YQzpceIAynqyILWAcMbOf4LsBpsm0GNrrTVcAdwB1unQVAJs6ojbXA+X44jgPd99/kxtJwLD3jFGCxe6y3ABP89G/eA+fHv5dHmd+PJ07yOgDU4Pw19m2c82MfALvc1xi37gTgWY9tv+V+R7OA2/wQZxZOv3TDd7RhBGES8PbpviPtHOdL7ndvM84PXWLjON3lU34X2jNOt/yFhu+kR12/HM/T/A61yffTbvdhjDGmRR2xG8oYY0wHY8nCGGNMiyxZGGOMaZElC2OMMS2yZGGMMaZFliyMMca0yJKFMcaYFv1/BMW/ia4m3TgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = log.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "For evaluation we first need to obtain survival estimates for the test set.\n",
    "This can be done with `model.predict_surv` which returns an array of survival estimates, or with `model.predict_surv_df` which returns the survival estimates as a dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "surv = model.predict_surv_df(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot the survival estimates for the first 5 individuals.\n",
    "Note that the time scale is correct because we have set `model.duration_index` to be the grid points.\n",
    "We have, however, only defined the survival estimates at the 10 times in our discretization grid, so, the survival estimates is a step function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEGCAYAAACO8lkDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAcpUlEQVR4nO3df3xU9Z3v8deHkBAloSgJigkIrvoQIooS0F7RxaorYBf2VlT8iWir3tVWr937qD/uVu0+ulqrbe2td6tVsbVd0er1ylWK2rXuD9cKqCgIoqxiGQQFXCXIj4Tkc/+YkzDJJGQS5uR7Zng/H495ZM6ZM+e8ORnyme/3nPM95u6IiIhk6hc6gIiIJI+Kg4iIZFFxEBGRLCoOIiKSRcVBRESy9A8doDeqqqp85MiRoWOIiBSM1157bZO7V+e6fEEWh5EjR7JkyZLQMURECoaZfdiT5WPtVjKzh8zsEzNb3sXrZmY/NbPVZvaWmR0fZx4REclN3MccHgam7OH1qcAR0eMK4B9iziMiIjmItTi4+78An+5hkRnArzztj8BgMxsWZyYREele6GMONcDajOlUNG99xwXN7ArSrQtGjBjRJ+FERDI1NTWRSqXYsWNH6ChdKi8vp7a2ltLS0r1aT+jiYJ3M63SwJ3e/H7gfoL6+XgNCiUifS6VSVFZWMnLkSMw6+/MVlruzefNmUqkUo0aN2qt1hb7OIQUMz5iuBT4KlEVEZI927NjBkCFDElkYAMyMIUOG5KVlE7o4zAcuic5aOhH43N2zupRERJIiqYWhVb7yxdqtZGaPApOBKjNLAbcApQDu/nNgATANWA1sA+bktOJN78Hcs2JIvBfGzoT63OKLiCRdrMXB3c/v5nUHru7pehs/beLDf0xQ71PjFwwa9xAHqDiISB9YuHAh1157Lc3NzXz961/nhhtuyPs2Qh+Q7p3S/eDgsaFTtNnx1muwYisHhA4iIkWvubmZq6++mhdeeIHa2lomTJjA9OnTGTNmTF63U5DFYdv+B/H6uOtCx2izo98Sahre4NDQQUSk6C1atIjDDz+cww47DIBZs2bx9NNPqzgAbG9qZsX6LaFjtKkqOwSvCJ1CRPrSbf/vbVZ8lN+/Q2MOGcQtf1m3x2XWrVvH8OG7T/Ksra3l1VdfzWsOKNTiMKAfbx0xIHSMNn/5b6s6vzhDRCTP0odq24vjDKqCLA6HVQ/ksSu/HDpGm9/821uhI4hIH+vuG35camtrWbt298ASqVSKQw45JO/bCX2dg4iI9MCECRN47733+OCDD2hsbGTevHlMnz4979spyJbDmi1rmLMwOaeNTup3IiXqVxKRPtC/f39+9rOfceaZZ9Lc3Mxll11GXV3+WzEFWRySpgWHhF81KSLFY9q0aUybNi3WbRRkcRg5aCRzp8wNHaPNg4/eFzqCiEhe6ZiDiIhkUXEQEZEsKg4iIpJFxUFERLKoOIiISBYVBxGRAnLZZZcxdOhQjj766Fi3o+IgIlJALr30UhYuXBj7dlQcREQKyCmnnMKBBx4Y+3YK8iI4EZHgfncDbFiW33UePBam3pHfdfaSWg4iIpJFLYd8cTjvvldCp2hnxrgaLjhhROgYIsUpId/w46KWQ74kbNy9Feu38PTSdaFjiEiBUsshj5J0A6KktWJEJD/OP/98XnrpJTZt2kRtbS233XYbl19+ed63o+IgIlJAHn300T7ZjopDnpQ2Os+dNTF0jDazGptZetSxQHJaMyJSOFQc8sD2L6GJ5tAx2hn+yTbgzdAxRKRAqTjkwQEVpVBRypm/XBQ6SpunzhgfOoKIFDCdrSQiIllUHEREJIu6lfJk07Zqnrr79dAx2mw65Cr2a3gjdAwRKVBqOeTBkVUrqdp/Y+gY7TSVHcL2yuNCxxCRPFu7di2nnnoqo0ePpq6ujnvuuSeW7ajlkAd1Q5dRN3QZzJkdOkqbX8z+RegIIhKD/v37c/fdd3P88cfT0NDA+PHjOeOMMxgzZkx+t5PXte3LNiyDuWeFTtGmH9PxpI3pISJ7bdiwYQwbNgyAyspKRo8ezbp161QcEmnszNAJOqHSIBKnHyz6Ae98+k5e13nUgUfxnYnfyXn5NWvW8MYbb3DCCSfkNQeoOORH/Zz0I0leui90AhGJ0datWzn77LP5yU9+wqBBg/K+/liLg5lNAe4BSoAH3P2ODq+PAH4JDI6WucHdF8SZSUQkH3ryDT/fmpqaOPvss7nwwgv52te+Fss2YjtbycxKgHuBqcAY4Hwz69gp9j+Bx939OGAW8L/jyiMiUgzcncsvv5zRo0dz/fXXx7adOE9lnQisdvf33b0RmAfM6LCMA63toS8BH8WYR0Sk4L388ss88sgjvPjii4wbN45x48axYEH+O1zi7FaqAdZmTKeAjkdNbgWeN7NvAgOB02PMIyJS8CZNmoS7x76dOFsOnZ0s0/FfdD7wsLvXAtOAR8ys00xmdoWZLTGzJRs3JuuCMxGRYhNncUgBwzOma8nuNroceBzA3V8ByoGqzlbm7ve7e72711dXV8cQV0REWsVZHBYDR5jZKDMrI33AeX6HZf4EnAZgZqNJFwc1C0REAoutOLj7LuAa4DlgJemzkt42s++Z2fRosW8D3zCzN4FHgUu9LzrTRERkj2K9ziG6ZmFBh3nfzXi+AjgpzgwiItJzGpVVRESyqDiIiBSQHTt2MHHiRI499ljq6uq45ZZbYtmOxlYSESkgAwYM4MUXX6SiooKmpiYmTZrE1KlTOfHEE/O6HbUcREQKiJlRUVEBpMdYampqwiz/YzCr5SAi0gsb/v7v2bkyv0N2Dxh9FAffdFO3yzU3NzN+/HhWr17N1VdfHcuQ3Wo5iIgUmJKSEpYuXUoqlWLRokUsX74879tQy0FEpBdy+YYft8GDBzN58mQWLlzI0Ucfndd1q+UgIlJANm7cyGeffQbA9u3b+f3vf89RRx2V9+2o5VDEWnDmLEzWHeqmHTaNc448J3QMkYK1fv16Zs+eTXNzMy0tLZx77rl89atfzft2VByKVImTHhd3w7LQUdqsohEaNqg4iOyFY445hjfeeCP27ag4FCmjP+WNu7j1N82ho7R5x5tYefRHoNogkngqDkXq8/2HMHD7Fjh4bOgobYa+tRje3hU6hojkQMWhSDVUDKahYjCH3vWr0FHavHNGXegIIpIjna0kIiJZVBxERCSLupWK2LaduzjvvldCx2gz251+MYwBIyL5p+JQpKoqytixvYX939sZOkqb9Ydcy8CtS0LHECl4zc3N1NfXU1NTwzPPPBPLNlQcitSk0w7l3UUfh47RzobPa9LXXojIXrnnnnsYPXo0W7ZsiW0bKg5Fqu7kGupOrgkdo50HZy8OHUGk4KVSKZ599lluvvlmfvSjH8W2HRUHEZFe+NfH32XT2q15XWfV8ApOPvfIPS5z3XXXceedd9LQ0JDXbXeks5VERArEM888w9ChQxk/fnzs21LLQUSkF7r7hh+Hl19+mfnz57NgwQJ27NjBli1buOiii/j1r3+d922p5SAiUiBuv/12UqkUa9asYd68eXzlK1+JpTCAioOIiHRC3UoiIgVo8uTJTJ48Obb1qzhIH3OYe1boEO2NnQn1ybopkkho6laSPuMYibsKbsMyWPZE6BQiiaOWg/QZx9IFYs6zoaPslrRWjCSeu2MJHiPM3fOyHrUcRERyVF5ezubNm/P2Bzjf3J3NmzdTXl6+1+tSy0FEJEe1tbWkUik2btwYOkqXysvLqa2t3ev1qDiIiOSotLSUUaNGhY7RJ9StJCIiWVQcREQki4qDiIhkUXEQEZEssRYHM5tiZqvMbLWZ3dDFMuea2Qoze9vM/jHOPCIikpvYzlYysxLgXuAMIAUsNrP57r4iY5kjgBuBk9z9P81saFx5JBmcFuYsTNBQFfYx03wg54TOIZIwcbYcJgKr3f19d28E5gEzOizzDeBed/9PAHf/JMY8EphRgiWsJ3MVjSywL0LHEEmcOK9zqAHWZkyngBM6LHMkgJm9DJQAt7r7ws5WZmZXAFcAjBgxIu9hJX5Gf4z+zJ0yN3SUNnMerg8dQSSR4vwa19ngIx2vOe8PHAFMBs4HHjCzwZ2tzN3vd/d6d6+vrq7Oa1AREWkvzuKQAoZnTNcCH3WyzNPu3uTuHwCrSBcLEREJKKduJTOrB04GDgG2A8uB37v7p3t422LgCDMbBawDZgEXdFjm/5JuMTxsZlWku5ne79G/QERE8m6PxcHMLgW+BXwAvEb6m305MAn4jpktB/7W3f/U8b3uvsvMrgGeI3084SF3f9vMvgcscff50Wt/YWYrgGbgf7j75rz960Ry0fhFMofu1k2IJKDuWg4DSZ9mur2zF81sHOluoKziAODuC4AFHeZ9N+O5A9dHD5G+NzA6fpW0EZg3LEv/VHGQQPZYHNz93q5eM7Myd1+a/0gifajy4PQjQWdQAclsycg+JacD0mb2kpmNzJieSPqYgoiIFKFcr3O4HVhoZj8lff3CVEDtXRGRIpVTcXD358zsKuAFYBNwnLtviDWZFKWm0iE8+o3kDKE1qekktlavhimhk4gkS67dSn8L/C/gFOBW4CUzU6eo9Mjn+zdS2pSsk9Ga+h9MxcbDQ8cQSZxcu5WqgInRWUuvmNlC4AHg2diSSdF5dVwdAI9d+eXASXZ78NIHaPHmZA0GCBoQUILLtVvp2g7TH5IebVWkR1as38J5970SOkabM7wfZkk7jzU9ICCGioMEE+fAeyLtzBhXEzpClpaWEkr6JWswQNCAgBKeioP0mQtOGMEFJyRrRN0H//290BFEEilZg+uLiEgidDe20ik5rmdNZ+MricheSOKYTxrvaZ/RXbdSrp+Cp+hifCUR6YUkjvmk8Z72Kd2NraRPgUgISRzzKWmtGImVjjmIiEgWFQcREcmi4iAiIllyvU3oI+5+cXfzRApRWeNOPrz4ktAx2pn16TusHF+lAQElmFwvgqvLnDCzEmB8/uOI9K0v9hsEbIGW0EnaG7puG+kBkEXC6O46hxuBm4D9zGxL62ygEbg/5mwisWuoGExDxWAOvetXoaO0885ZE0NHkH3cHo85uPvt7l4J/NDdB0WPSncf4u439lFGERHpY3ssDq23Bu2qEFhabf5jiYhISN0dc/ihmfUDngZeAzYC5cDhwKnAacAtQCrOkCIi0re6u0L6HDMbA1wIXAYMA7YDK0nf6Of77r4j9pQiItKnuj1byd1XADf3QRaRILbt3JWoGxABzGpsprRElyFJON2drTQBWOvuG6LpS4CzgQ+BW9390/gjisSnqqIskSeMNrc4LWxP1u1LdevSfUp3LYf7gNOhbfjuO4BvAuNIn8o6M9Z0IjEbWlnO0Mpybr7y+NBR2pn/20E00xA6Rju6dem+pbviUJLROjgPuN/dnwSeNLOl8UYT2XeVMphSH5yo25fq1qX7lu46NUvMrLWAnAa8mPGabjEqIlKkuvsD/yjwz2a2ifRZSv8KYGaHA5/HnE1ERALp7lTW75vZP5E+hfV5d2+9L1U/0sceRESkCOVyKusfO5n3bjxxRPreptRWnrr79dAx2vm05mtUNKwKHUP2YTpuIPu0IyceFDpCpxrLqtlaGTqF7MtUHGSfVndyDXUn14SOkeXB2ckaJVb2PboEU0REsqjlICK5a/wC5p4VOkV7Y2dCfYKuJC8SsbYczGyKma0ys9VmdsMelptpZm5muspGJKkGVkPZwNAp2tuwDJY9ETpFUYqt5RDdSvRe4AzSQ3ovNrP50UB+mctVAt8CXo0ri4jkQeXB6UeCrtpOXCumiMTZcpgIrHb39929EZgHzOhkub8D7gQ09LeISELEWRxqgLUZ06loXhszOw4Y7u7PdLcyM7vCzJaY2ZKNGzfmN6mIiLQTZ3GwTuZ524vpO8z9GPh2Litz9/vdvd7d66urq/MUUUREOhPn2UopYHjGdC3wUcZ0JXA08JKZARwMzDez6e6+JMZcIgWhrHEnH158SegYbWZ9+g4rx1fBlNBJpC/EWRwWA0eY2ShgHTALuKD1RXf/HKhqnTazl4C/UWEQgS/2GwRsgZbQSXYbum4bJPLWSBKH2IqDu+8ys2uA54AS4CF3f9vMvgcscff5cW1bpNA1VAymoWIwh96VnCul3zlrYugI0odivQjO3RcACzrM+24Xy06OM4uIiOROw2eIiEgWFQcREcmi4iAiIllUHEREJIuKg4iIZNGQ3SIJtW3nLs6775XQMdrMamymtETfJ/cVKg4iCVRVUZa4y82aW5xEXZUnsVJxEEmgoZXlDK0s5+Yrjw8dpc1TT3Q2XJoUKxUHEclZCzuZszBBd12zj5nmAzkndI4ipOIgklCbUlt56u7XQ8dos/mQqylvWAJsDx2lzSoawVBxiIGKg0gCHTnxoNARsjSVDcMqT2TulOSMFDvnYd1ZOC4qDiIJVHdyDXUn13S/YB96cHZyBgGU+Om8NBERyaLiICIiWVQcREQki4qDiIhkUXEQEZEsKg4iIpJFxUFERLLoOgcRyVlZ404+vDg5F8HN2rCdlXX6MxYH7VURyckX+w0CtiRqYNahn7QAu0LHKEoqDiKSk4aKwTRUDObQu5JzpfQ7Z9SBt8Dcs0JHaW/sTKhP0ACFvaBjDiJSuEpKwRL2Z2zDMlj2ROgUe00tBxEpXCVl6cecZ0Mn2S1prZheUnEQkZzp1qX7DhUHEclJVUUZO7a3sP97O0NHabNx2FXst3WxbkAUAxUHEcnJpNMO5d1FH4eO0c76z2vYYQD/HjpKm2K5AZGKg4jkJLn3mBjA3ClzQ0dpUyw3IFJnnYiIZFFxEBGRLCoOIiKSRcVBRESyqDiIiEgWFQcREcmi4iAiIllivc7BzKYA9wAlwAPufkeH168Hvk56zN2NwGXu/mGcmUSkuDS3eKKG9HDbVRRDesT2LzCzEuBeYCowBjjfzMZ0WOwNoN7djwGeAO6MK4+IFJ/Skn6U9LPQMdppaXGamhN004teirPlMBFY7e7vA5jZPGAGsKJ1AXf/Q8byfwQuijGPiBSZ0pJ+lJb047Ervxw6Sptz709WseqtONs+NcDajOlUNK8rlwO/6+pFM7vCzJaY2ZKNGzfmKaKIiHQmzuLQWfn0Thc0uwioB37Y1crc/X53r3f3+urq6jxFFBGRzsTZrZQChmdM1wIfdVzIzE4Hbgb+3N2TMxawiBSElm3b+PDiS0LHaHPp+kZeqrNkDSPeC3G2HBYDR5jZKDMrA2YB8zMXMLPjgPuA6e7+SYxZRKQI9R8yhH777x86Rju1nziT3+60kySYVZ+u6vF7Yms5uPsuM7sGeI70qawPufvbZvY9YIm7zyfdjVQB/NbMAP7k7tPjyiQixaV/dTX9q6s59NsXhI7SZsWfj2VkE8kaRnzhHF6hZ6f7xnqdg7svABZ0mPfdjOenx7l9ERHpHd3sR0QK2qbUVp66+/XQMdpsG3ENB215jY4XdRUaFQcRKVhHTjwodIQsW8uTdbe83lJxEJGClcRbl/7m4sWhI+RF4Q8AIiIieaeWg4hInvWjBeaeFTrGbvZxj9+iloOISB7tsv60JO1Pa+MXPX6LWg4iInm0y0rZZaUw59nQUXZ7uL7Hb1FxEBHJsyTeY6KnVBxERPIofaOfZN3PoaWl58N5qDiIiORRaUk/tvc7gL9a8kHoKG22Nf41T3JVj96TsKMmIiKF7fAxAxlkn4eO0U5Tac+vBVHLQUQkjyb89xlMCB2igwdn39fj96jlICIiWVQcREQki4qDiIhkUXEQEZEsKg4iIpJFxUFERLKoOIiIFLmynakev0fFQUSkyFVterLH71FxEBEpchVW0uP3qDiIiBS5EWVf6vF7VBxERIrcgJOm9fg9Kg4iIkXu4Jtu6vF7VBxERCSLioOIiGRRcRARkSwqDiIikkXFQUREsqg4iIhIFhUHERHJouIgIiJZzN1DZ+gxM2sAVoXO0UtVwKbQIXpJ2cNQ9jCKLfuh7l6d6wr65zdPn1nl7vWhQ/SGmS1R9r6n7GEoexj5yK5uJRERyaLiICIiWQq1ONwfOsBeUPYwlD0MZQ9jr7MX5AFpERGJV6G2HEREJEYqDiIikqWgioOZTTGzVWa22sxuCJ2nO2a2xsyWmdlSM1sSzTvQzF4ws/einweEztnKzB4ys0/MbHnGvE7zWtpPo9/FW2Z2fLjkXWa/1czWRft/qZlNy3jtxij7KjM7M0xqMLPhZvYHM1tpZm+b2bXR/MTv9z1kT/x+j7KUm9kiM3szyn9bNH+Umb0a7fvHzKwsmj8gml4dvT4ygdkfNrMPMvb9uGh+zz837l4QD6AE+A/gMKAMeBMYEzpXN5nXAFUd5t0J3BA9vwH4QeicGdlOAY4HlneXF5gG/A4w4ETg1QRmvxX4m06WHRN9fgYAo6LPVUmg3MOA46PnlcC7Ub7E7/c9ZE/8fo/yGFARPS8FXo326ePArGj+z4H/Fj3/a+Dn0fNZwGMJzP4wMLOT5Xv8uSmklsNEYLW7v+/ujcA8YEbgTL0xA/hl9PyXwF8FzNKOu/8L8GmH2V3lnQH8ytP+CAw2s2F9kzRbF9m7MgOY5+473f0DYDXpz1efc/f17v569LwBWAnUUAD7fQ/Zu5KY/Q4Q7cOt0WRp9HDgK8AT0fyO+771d/IEcJqZWR/FbWcP2bvS489NIRWHGmBtxnSKPX8Qk8CB583sNTO7Ipp3kLuvh/R/LmBosHS56Spvofw+roma0Q9ldOElMnvUTXEc6W+BBbXfO2SHAtnvZlZiZkuBT4AXSLdmPnP3XdEimRnb8kevfw4M6dvEu3XM7u6t+/770b7/sZkNiOb1eN8XUnHorEIn/Tzck9z9eGAqcLWZnRI6UB4Vwu/jH4A/A8YB64G7o/mJy25mFcCTwHXuvmVPi3YyL2nZC2a/u3uzu48Dakm3YkZ3tlj0M1H5O2Y3s6OBG4GjgAnAgcB3osV7nL2QikMKGJ4xXQt8FChLTtz9o+jnJ8BTpD98H7c256Kfn4RLmJOu8ib+9+HuH0f/gVqAX7C7CyNR2c2slPQf19+4+/+JZhfEfu8se6Hs90zu/hnwEun++MFm1jruXGbGtvzR618i967M2GRknxJ19bm77wTmshf7vpCKw2LgiOhMgjLSB4TmB87UJTMbaGaVrc+BvwCWk848O1psNvB0mIQ56yrvfOCS6CyIE4HPW7tBkqJDn+p/Jb3/IZ19VnT2ySjgCGBRX+eD9FkkwIPASnf/UcZLid/vXWUvhP0OYGbVZjY4er4fcDrp4yZ/AGZGi3Xc962/k5nAix4d7e1rXWR/J+MLhZE+VpK573v2uQl1tL03D9JH3N8l3S94c+g83WQ9jPSZGW8Cb7fmJd1H+U/Ae9HPA0Nnzcj8KOlugCbS3zQu7yov6WbqvdHvYhlQn8Dsj0TZ3or+cwzLWP7mKPsqYGrA3JNIN+/fApZGj2mFsN/3kD3x+z3KcgzwRpRzOfDdaP5hpIvWauC3wIBofnk0vTp6/bAEZn8x2vfLgV+z+4ymHn9uNHyGiIhkKaRuJRER6SMqDiIikkXFQUREsqg4iIhIFhUHERHJ0r/7RUT2XWbWekopwMFAM7Axmt7m7v8lSDCRmOlUVpEcmdmtwFZ3vyt0FpG4qVtJpJfMbGv0c7KZ/bOZPW5m75rZHWZ2YTTe/jIz+7NouWoze9LMFkePk8L+C0S6puIgkh/HAtcCY4GLgSPdfSLwAPDNaJl7gB+7+wTg7Og1kUTSMQeR/Fjs0Vg1ZvYfwPPR/GXAqdHz04ExGbcAGGRmlZ6+F4JIoqg4iOTHzoznLRnTLez+f9YP+LK7b+/LYCK9oW4lkb7zPHBN60Tr/X1FkkjFQaTvfAuoj+7StQK4KnQgka7oVFYREcmiloOIiGRRcRARkSwqDiIikkXFQUREsqg4iIhIFhUHERHJouIgIiJZ/j9Au0ub7sJ4/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "surv.iloc[:, :5].plot(drawstyle='steps-post')\n",
    "plt.ylabel('S(t | x)')\n",
    "_ = plt.xlabel('Time')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is, therefore, often beneficial to interpolate the survival estimates, see [this paper](https://arxiv.org/abs/1910.06724) for a discussion.\n",
    "Linear interpolation (constant density interpolation) can be performed with the `interpolate` method. We also need to choose how many points we want to replace each grid point with. Her we will use 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "surv = model.interpolate(10).predict_surv_df(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEGCAYAAACO8lkDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXyU5b338c+VsAQIkSWsCatsBlHEAFqX4lYRPaVWEbQVxbbY82hrFz3Vek6P7XOsy9G2nravHm0rLq0b7fGRQ1Fbq7RWWwQVWSIBZJEAIQkRkhCyX88f98zkzp3JTCbMPUvyfb9evMi9zD2Xk8gvv9+1GWstIiIibhnJboCIiKQeBQcREWlHwUFERNpRcBARkXYUHEREpJ1eyW5AV+Tm5trx48cnuxkiImnj3XffrbDWDuvs/WkZHMaPH8+GDRuS3QwRkbRhjNkby/2+lpWMMY8bY8qMMVs6uG6MMf9ljNlpjNlkjJnlZ3tERKRz/O5zeAKYH+H6ZcDkwJ/lwC98bo+IiHSCr8HBWvtXoDLCLQuBp6zjH8AgY8woP9skIiLRJbvPIQ/Y5zouCZw76L3RGLMcJ7tg7NixCWmciIhbY2MjJSUl1NXVJbspHcrKyiI/P5/evXuf0HOSHRxMmHNhF3uy1j4GPAZQWFioBaFEJOFKSkoYOHAg48ePx5hw/3wll7WWw4cPU1JSwoQJE07oWcme51ACjHEd5wMHktQWEZGI6urqGDp0aEoGBgBjDEOHDo1LZpPs4LAKWBoYtXQWcNRa266kJCKSKlI1MATFq32+lpWMMc8C84BcY0wJ8O9AbwBr7X8Da4AFwE6gFljWqQdX7IAVlztfz7gaCjv3MhER6Rxfg4O19too1y1wS6zPbahsZO8zB6DhGDkzH2ewgoOI9CCvvPIKt912G83NzXz5y1/mzjvvjPt7JLus1DW9+8HIGdQd6U1VUU2yWyMikjDNzc3ccsstvPzyyxQVFfHss89SVFQU9/dJ9milLqntP4L3Zn6DuowNDD+wHq5fGrqWc8UVDF58TRJbJyLin3feeYdJkyYxceJEAJYsWcJLL71EQUFBXN8nLYPD8cZmig5WMTQrD0Y2M6n0MQDqyhqgulTBQUR89/3/3UrRgaq4PrNgdA7//k/TI96zf/9+xoxpHeSZn5/PunXr4toOSNfg0DeDTZP7MndzOWSP5c2CWwHIHNnAyJoPGJfk9omI+MXpqm3LjxFUaRkcJg4bwPM3n82KpzZzYNNhDnMyAEOzamhpymCvykwi4rNov+H7JT8/n337WheWKCkpYfTo0XF/n7QMDkHLls5oc/yb5b/E9Grh2MfvOycqm6kq36fgICLdxuzZs9mxYwe7d+8mLy+P5557jmeeeSbu75OWwWFP1R6WveIMX10wcQGLpiwCoHnIcGqO9uK9md8AIKOygeEH3yMnkEkoixCRdNerVy9+9rOfcemll9Lc3MxNN93E9Onxz2LSMjgEFVcWA4SCQ+HlhWx/5xDgLOx6qO4gLaMyaDz4MiPLP+ZQdT1zFBxEJM0tWLCABQsW+PoeaRkcxueMZ8X8FSx7ZRnFlcWhLAJgwT+1ZhLPf2sFmX2bGHtKBfblBuqOatkmEZHOSMvgELRgYtvIueHQBjYc2sCaXWsAmNrn0+TUTyDzk69QN76WYQff5eWLPxe6P+Pi+Vx651cT2mYRkXSQ1sFh0ZRFoSwBYOX2laHAAFA07D0KMs9i1JAZVFU5JaYhB18GILd0L7v+939ZPPj00P0LZ+Zx3VztFSEiktbBwcsbLJaxjFWVP2PqkKlMLf0UZkALX7iwEYAdT7dQULGPnJUPAHCsvoldm86BuXclpe0iIqmkWwUHL3fZqdZAv/qxvFjkdEg3TaplxKBNFAw4BkDlpi2w6S0WP/r30GuUSYhIT9Wtg4M7k7hjzw85vDuD4736AdCv91g+Obkv5//nTQAc+txiTt69gxuVSYiIdO/g4Dbnwslt+iPG/Pls+rt2KZ187VVUrV7NkMCxN5NQFiEiqeCmm25i9erVDB8+nC1btvj2Pj0mOHj7I+7/8y/pWzuK++/4Zejc6HPPYOnN3wScTGJATT0ARQedxbUUHEQk2W688UZuvfVWli5dGv3mE9BjgoPX6Ek1HNjZuiNpn9pRFH+0PzRnYknjbsZ80siDf/sFRQereHXk6Sx2vV6ZhIgkw/nnn8+ePXt8f58eGxyCGULQY3c8TuaxfAa+eQkAW8Z9iqMDPmAuVeRXlnAp8ATzAVi3u5J1uyt5aeP+0OsVLER6mJfvhNLN8X3myBlw2f3xfWYX9djg4HVO7na2H6iDhgEAfGxGs2U8vHzh2yz5aQZjSvfy4N9+AcCh6nrW5s3kXeYBKjuJSPej4BAw/aKpTN/8u9DxY5uvoL91Pp43pjVzAb2ZFrh20v7dLB7Yl3+52RnJtPjRv1N0sEqd1yI9SYr8hu8XBYegwmXOn4BhdzyJqR3GZ7d+jeKsbfz2wn+QOb0agCU/haHHy0ObCi2cmRd6nbIIEekOMpLdgFQ1JfdDcnvtgdLN5BwZREH5rNC16sZq2LGHVy+fw6uXz2HoD67mK/X/j+dvPpuCUTnJa7SIdHvXXnstZ599NsXFxeTn5/PrX//al/dR5tABd5npxY2XM6p5AHfOXwHAa9u/w+FX3wjdm1tSQ8Wrb8DXk9JUEelBnn322YS8j4JDR9xlpjuepKJ2GC8+/F7g4mKm3Pl1pp/nlJNevXwOuSU1vHr5HJY0NPOXcQUa9ioiaU1lpU6Ykvshuf3LQ8cVJTWBTYUcmZdeQEV+NgBjymr59N6i0LWig1VthryKiKQDZQ6dMH34Zqa3PANDnD2rXyy9BqqHha5f/PUHQiWlVy+fQ3/g+ZvPBmizkJ+ISLpQcOiMGVe3PW445ikzwZQ5I0JlpmCJCWBJQzMbp50OnJ2o1oqInDAFh87wDHOd8sCdUDEAGAk4ZSaA6eflOSUmV2f1mLJamls2ailwEUkrCg5dEK7MVLE3L5BJtO2sXj1/NpnNLaHXah6EiKQDBYeu8JSZpvR+DbgYGNImiwDonWkYdfA4N626DYBalZlE5ATs27ePpUuXUlpaSkZGBsuXL+e2226L+/soOHSFp8w0fcXlTOcFWHZDm34IIGyZCT5IVEtFpJvp1asXDz/8MLNmzaK6upozzzyTSy65hIKCgvi+T1yf1pOVboYVl0PpNTAg/EgmgBcvOZPmFqs+CBHpklGjRjFq1CgABg4cyCmnnML+/fsVHFKSu8wUZSRT78wMxpTVsOR3twLQ3GLZvGUmzH08oU0WkRPzwDsPsK1yW1yfOW3INL4z5zudvn/Pnj28//77zJ07N67tAAWH+HCVmaY8cCccAEqdSXMVtcOgujQUHLIWXMjhV9+gf+ClQ0uqydymMpOIxKampoarrrqKn/zkJ+TkxH9NN1+DgzFmPvAIkAn8ylp7v+f6WOBJYFDgnjuttWvaPSiNeJf+fnHj5XBsQOg4XJmphfrQDnQLJi5os52piKSmWH7Dj7fGxkauuuoqvvCFL/D5z3/el/fwLTgYYzKBnwOXACXAemPMKmttkeu2fwVesNb+whhTAKwBxvvVpoTwdFZzx5NU1AzlxTueDJ2acnp/pn/RCQCZdiBjyur47MObaKGerWfshwcVHEQkPGstX/rSlzjllFP41re+5dv7+Lm20hxgp7V2l7W2AXgOWOi5xwLBfOgknIJMtzLl9P7kZh8OHVfUDGX7B7Wh474XL6F8+GSy7BjGHbLMeP9wuMeIiADw1ltv8fTTT/P6668zc+ZMZs6cyZo18S+4+FlWygP2uY5LAG+vyT3AH40xXwMG4EwW6Famf3ER07/YevziHU9CwzFnZBNw6Qjg/quhcFm7EhOozCQibZ177rlYa31/Hz+DgwlzzvtfdC3whLX2YWPM2cDTxphTrbUt3hcaY5YDywHGjk3jYZ8DhjmjmYqucY4bjjGlrJjphU6JaWx5HUt+uhWA2qbj7J17CH6g4CAiieVncCgBxriO82lfNvoSMB/AWvt3Y0wWkAuUeR9mrX0MeAygsLDQ/7DpkykXnQ7vHCK0LtOuUqgYwHRg29RLaah/iwH1zrdlbPkOer9T3vHDRER84mdwWA9MNsZMAPYDS4DrPPd8DFwEPGGMOQXIArr1v4bTz8sLDWuFtmWmG6bW8dKkT7Gy/wIAFr1wC6b5uMpMIpJwvgUHa22TMeZW4FWcYaqPW2u3GmN+AGyw1q4Cvg380hjzTZyS0402EcW0VOIuMzUc45zRH7P8m866S6tW5pBXVq4yk4gknK/zHAJzFtZ4zn3P9XURcI6fbUh17jJTxa5SOOCs1QRwcn4NVc1DmDZkAgCVm9+j/7sVSWytiPQUmiGdZO4y04v3rGkzYW78hAMUjR/Hv4z6ZwCWlN6KbaxVmUlEfKfgkEoGjqTiaDYvVv5fABqOfsDk3q/xvcN3APBJUx39K63KTCI9WF1dHeeffz719fU0NTVx9dVX8/3vfz/u76PgkEKmzBnR5riqaQx7Mz/DlaNeAOBQ/j4Ot+Qwbcg0QGUmkZ6ob9++vP7662RnZ9PY2Mi5557LZZddxllnnRXX91FwSCHtRjI9/B4wBJbdAEDFwXMZc/JH7OuzC4CDB1uoazymdZlEehBjDNnZ2YCzxlJjYyPGhJtWdmIUHFJcRUlNaPnvqqo7OJDxOqP7vAtAbnMTvcozWfLTrSoxiSRY6Q9/SP2H8V2yu+8p0xj53e9Gva+5uZkzzzyTnTt3csstt2jJ7p7GW2ZqqB9MTf4NTP/2IwBs3V5Iv+ajDG1opLKsif7vdLulqUQkjMzMTDZu3MiRI0e48sor2bJlC6eeempc30PBIYWFLzO1euv0yzhnyhtMH3US2367i9rmBo1kEkmQzvyG77dBgwYxb948XnnlFQWHns5dZio/eBHfzzgPM3QgX2/+CsMrm1jy8DsA1GLZe8YOLf8t0s2Ul5fTu3dvBg0axPHjx3nttdf4znfiv7eEgkMa8ZaZBjfA6X36sAnYlVfAmS0fMi3wLa0sq4P3K5VJiHQzBw8e5IYbbqC5uZmWlhauueYarrjiiri/j4JDGglXZhoF3H3zLBYDq4Hnb3aW3ii/5DT620Yo3QxAMQ1QXargIJLmTjvtNN5//33f30fBIc0Fy0ynHaynbHBm6PywASMZuHs/9/y2GYBtLQ18OO0jqHGW5mDG1W13rBMRcVFwSGPuMlP28RZq65tY/OjfATgz7wLmsZGsgX0BGL55A7XbM1g2+xA0HGPB5idYpOAgIh1QcEhj7jLTo/e8Tf+ahtC1p4fN4t3T5rWWma6+jP7HD8PI6RSXvgsNZSwKLPAHKJMQkTYUHLqJ4QOzyDjaxJIaJ1MoqulLWUVT6PqwfsMY1m8YK+avYNnKy4Dy1n35Av0SCg4iEqTg0E14RzJlH2+30yp127ax9/qlLKk8zBvTLMsuGe5cMMNZcHSbMgkRCVFw6Ca8I5nuvX1t2z6InFOYl1dPFpBX2sgF9Oa5S5x7izOBkwazKJhJ7P2b82fz71rfQMFCpEdRcOimcrP7UHe8hf476gE4mjGb384+l0fuPo+91y9lGrBi/gqA1rkQgWM2rGgbGFR2Ekkpzc3NFBYWkpeXx+rVq315DwWHburci8ax/Z1DoeOPdx2h/yfNoeNgiQlgSeU23pjWzDJcE+bOubF1TsSKy50AsULDYEVSwSOPPMIpp5xCVVWVb++h4NBNhSszBeV4ZlO2KzNVFgO0BocZV7ferJKTSFKVlJTwhz/8gbvvvpsf/ehHvr2PgkMP0u9YcyBIDGf0JXewbOkMgLBlpuLK4rZLbwQzCZWcRAB484XtVOyrieszc8dkc941UyLe841vfIMHH3yQ6urquL63l4JDDzH6tKEc2HQYcIJE8OtwFkxc0Oa4TSZRuKxtIPCWnECZhIhPVq9ezfDhwznzzDNZu3atr++l4NBDBLMEaFtiCnL3QcwBLr7iCgYvvsZ57SsR/qF3l5ygfdlJgUK6qWi/4fvhrbfeYtWqVaxZs4a6ujqqqqr44he/yG9+85u4v5eCQw/VWmKCgQM+z8RJG5mBs/1o3TZnd6tgcADalJnarO7qzSTcZSeVnETi6r777uO+++4DYO3atTz00EO+BAZQcOiR3CUmgCYzkA/y5nHFQ/cAhDKIIHeZqV1ntZc7WKjkJJK2FBx6IHeJCcKXmdwWTVkUCgYRS0xe3pKTMgmRuJk3bx7z5s3z7fkKDgI4y20Ed5iry5rP8JK3wZVB5Lj6INqNZOpoEyF1XoukLQUHoWxwJrX1TdQcdCbUnJSZS+Oos5jU+EegbR9ExJFM0UTrvA7eo2AhknQKDsLci8fx0sb9oeNJW5o4mp3PuIeeAtr2QbhLTBBjmSlS5zWo7CRpwVqLMSbZzeiQtTb6TZ1g4vWgRCosLLQbNmxIdjO6rXtvX0v28RbGThwEOJnD8JK3mZTrZBY5nmGuxZXFTB0yNfT6Lu9VHSw7jQz0iSiLkBSze/duBg4cyNChQ1MyQFhrOXz4MNXV1UyYMKHNNWPMu9baws4+S5mDtBOpzOQd5npCZSYvd9lJWYSkoPz8fEpKSigvL092UzqUlZVFfn7+CT9HmYO088y6jz1lpmP079uLux+aFyoxjXv6qbCv9WYSccsiQJmEyAlQ5iAn7Lq5Y7lu7tjQcbDM9OLD71GXNZ+8pl2M6+C1Mc2JiETDYEWSSsFBoiobnBn6uipjCC0NtW06qd19EF2eE+GlYbAiSaXgIFGV5vbi9cZaCrLrOcfU0a9vFjQ618ItteHW6TkR0WgNJ5GEUnCQqBbObN0X4lDvbGxmDu9N/wYAdVnbOiwzxbWzWms4iSSUrx3Sxpj5wCNAJvAra+39Ye65BrgHsMAH1trroj1XHdLJc9u9bzL8k2YKRuUAULa9jJyWSq79ZdRvWyiDCO4bETfqvBaJKmU6pI0xmcDPgUuAEmC9MWaVtbbIdc9k4C7gHGvtJ8aY4X61R+LDXWIC+ExdGc2N9R32QXjFrczkps5rkbjzs6w0B9hprd0FYIx5DlgIFLnu+Qrwc2vtJwDW2jIf2yNx4C4xAVT2ySa7zzDeZj4ALbW1jHllO+cvbv9ab5lpw6ENbDi0gTW71rS5J+Zgoc5rkbjzMzjkAftcxyXAXM89UwCMMW/hlJ7usda+Eu5hxpjlwHKAsWPHhrtFEsA7zPW2CqfMNHLUCMApM+1v6R/2td6lN1ZuX9kmMJxQn4SbMgmRE+Zbn4MxZhFwqbX2y4Hj64E51tqvue5ZjTPu5RogH3gTONVaeyTSs9XnkDoWP/p3AJ6/+WwAnv3KM7TU1nJuxl+ByCUmr7hNoPNSn4RI6vQ54GQKY1zH+cCBMPf8w1rbCOw2xhQDk4H1PrZL4qzoYFUoSJyT2Z8+Awbztp0fscQUjrvsFLeSEyiTEOmCTmUOxphC4DxgNHAc2AK8Zq2tjPCaXsB24CJgP84/+NdZa7e67pkPXGutvcEYkwu8D8y01h4O98wgZQ6pw7vUhv2omtNb+lAwKiemkUxe3pLThkPO97twROsvPlrgT6Tz4po5GGNuBL4O7AbeBYqBLOBc4DvGmC3Av1lrP/a+1lrbZIy5FXgVpz/hcWvtVmPMD4AN1tpVgWufMcYUAc3AHdECg6QWbx/E4kf/zibg7ptn8exXnunyc33tn3BnEtpTQiSsiJmDMeYWnH/Uj3dwfSYw1Fr7Z5/aF5Yyh9S1+NG/U3SwioJROXzm7R30baznvMw3Q9dj6YOIJG79E949Jfb+zfl73Lmt5xQspBuIa+Zgrf15hDfqY63dGEvjpPtzD3WNZZhrrOLWP6ENiETC6myfw1rgRmvtnsDxHOCX1trTfW1dB5Q5pIcTmU0di4T2T4AyCUlLfo1Wug94xRjzXzjzFy4D9H+HRFSa24vS3F7cffMsgBPqg4gkYf0ToAX/pMfo9DwHY8w84E9ABXCGtbbUx3ZFpMwhPbj7HwA+8/YOGnsP5SRzNHTPpIIBzP7mQl/bEdetTN1lJ/VPSBrxJXMwxvwbzkS184HTgLXGmG9ba//QtWZKT+BdamOvqWVcI9DH+bGrsjlsX1fC8E6uy9RV0ZbtiClQuPsownVma+STdBOd7XN4BLgzOGrJGDMOZ5XVS3xuX1jKHNKTdzb1yjtfpfITS06LM12mpbqKEYc2MDm/AfAnUEDbslNc+yeijXxSoJAkijVz0B7SkjDeMtPIiiYKGjMZPjALgPLdlQysL+NTda9Qt20bWdOmdbhXdbyE65+YOmRqfJYVVwlKUoiCg6Qs72zqYKAIZhIvPvweAFd+exZ7r18aChBBfmUSbnHtn3DTfApJMgUHSRveTOK0HfUMboBR43JoKi9nxKH1jGvaDkDteme5rf6zZ4de70ew8DWTcAs3n2LkDFimbjzxh4KDpI1I6zJVlNSQm5/Nld92hsF+8vwLVK1eHbo3UWUn3zIJL82nEJ/Fe22l8zv5nD3h1lcSiSTcukzPHayiILue0zJbaKmuC10bvPiaNllCsOwU3IHOr5JTXPfBjkTzKSTFRFtbqbO59IuBhfQSQplD9+TOJMZvqmF4cwZ12Zmh66NPG8qypc5v1u5MIlElJ/Bxzwkvd9lJJSeJA5WVpFtY8dRmDmxqXaC337Fmjg/I5O6H5rW711ty8jNYuPskfOuP8FLJSeJAwUG6pXtvX0ttfRM7Tx0QOrdwZl6bslRQovonEtYfoc5riYNU2glOJG5ys/tQd7yF/jvqAaitb2Jdxd6wwSFa/wTEJ5NIWH+Ed+XYFZfH9/kiYShzkLSw9c39bH/nUOj4411HKMtoCWUSHWUREL3s5NceE6CRTZI6/Fpb6Wlr7fXRzon4Zfp5eUw/r3WtpkfveZvh5XX031EfMYuA9pmEtzO7dv36NsGjq8EiaSObtOeE+KCzayu9Z62d5TrOBDZbawv8bFxHlDmIO5PwZhEQOZNw87N/ImEjm7QntnRCvOc53AV8F+hnjKkKngYagMe63EqRE+TOJB6952361zSErq3bXcm63ZVtJth1FCzC9U/EizuT8C2LgLaZhLIIiZPOZg73WWvvSkB7OkWZg7i9+PB7oRnVAGXVdRT1bqY01/ndx7uGUyR+remk/ghJtnhnDuOttXs6CgzGGAPkWWtLYmynSNxMmTOizXHG0Sbm5WdzZWAHuuAaTsElw6HjTCLniivaHNdt2waQPiOb1B8hcRJthvRKIAN4CXgXKAeygEnABcBFwL9ba//kf1NbKXOQSNyru0L01WAj6RaZBGhOhMQ3c7DWLjLGFABfAG4CRgHHgQ+BPwD3WmvrIjxCJCkqSmpCQaIf8L05Y0N9FN5MIlLndaIyCe/udMF74hIsSje3nRuhMpN0QtShrNbaIuDuBLRFJC68ZaaKkhqAUHBwb18arfParwX/Fk1Z1OYf/nBLhQfvOyEqM0kXRSsrzQb2WWtLA8dLgauAvcA91trKhLTSQ2UliYW3zOQWa8nJPfTVz2XDfRsGqw7rHivek+AeBS4OPPh84H7ga8BMnKGsV3f8UpHU4S4zgZNdTD8vL+yy4ZG4M4l4Dnv18m0YrDIJ6aRowSHTlR0sBh6z1v4e+L0xZqO/TROJj2hlJq/OjmwCfFmzCdqWnZa9Esd/uLVOk3RS1OBgjOllrW3CGZm0PIbXiqQE79IbwXkRwUwimEVA2/4IcAIFkNBhr+EUVxa3CRJxHdnk7rBWiUkCov0D/yzwF2NMBc4opTcBjDGTgKM+t03EF+5MwptFhCszdZRJdIvVXzW7WjoQbSjrvcaYP+MMYf2jbe29zsDpexBJO+5Mwt0PEU4qZBLekU2+lZmCndUa9ip0bijrP8Kc2+5Pc0QSr6POagifSXQkkftc+1JmUme1uKjfQHq0WDurgS5NoItnf4RvZaZwndXKJHosBQfp0aJ1VkPHHdaRSkzg37BXX8tMbsokejQFBxGXaJmEu8wUy4J+4N+wV2hbZorbSCYNe+3RFBxEXMJlEh1Jhc5qSOC+EaBhrz2Ir3tIG2PmA48AmcCvrLX3d3Df1cBKYLa1Nuq6GFo+QxLFu1cEtC0zuQUziG672uuGFbD5d87XwSU4tNpr2vBlD+kuNiQT+DlwCVACrDfGrAos5Oe+byDwdWCdX20R6Sq/Zlen5b4R3mGv0q35ljkYY87GWZzv0sDxXQDW2vs89/0EeA24HbhdmYOkMm8m4c4iUnXfCIAV81ec0HPa0QJ+aSdlMgcgD9jnOi4B5rpvMMacAYyx1q42xtwe6WHGmOUElu8YOzb6xvEifjiR2dXJ2DcCNCdCusbP4GDCnAulKcaYDODHwI2deZi19jGclWApLCz0r6NEJALv7Go/hr1C/Ia+JnROhHQrfgaHEmCM6zgfOOA6HgicCqx1tqJmJLDKGPPZzpSWRJIt1mGvsfJjUyHf5kSAJsx1M34Gh/XAZGPMBGA/sAS4LnjRWnsUyA0eG2PW0sk+B5FUEMuwV4htKXC/ZleDykzSOb4FB2ttkzHmVuBVnKGsj1trtxpjfgBssNau8uu9RZKlozJTLHMioP3s6nhNoFOZSTrL10lw1to1wBrPue91cO88P9si4rdIZaZYd5xzi2dntcpM0lm+ToLzi4aySjqINOw1OJKpYFRO6P5IZSa3eA579W2vaveEOdCkuRSQSkNZRXq0SMNeYy0zucUzk/Bt6Q2VmdKeMgeRBIi2DEc8M4l4ZREQx0zCO2lOJaaEU+YgkoKiDXuNVyYRrywCtB1pT6fMQSQJgiOarvz2rLDXY1nEzy04omnc00+dQOscvi69Aep/SDBlDiJpItLsauj8jnNe8dw3wpc5EZIWFBxEkiCWMlMqdFZDnMtMGuaa8lRWEkkBkTqsU2nYK8ShzKRhrkmhspJIGoqUSXg7q9ftrmTd7so2y4Mnat+IuGxHqmGuaUGZg0gKSsV9I1ZuX8maXc6CBxsOOf//FY5o/UW0y8FCe0MkhDIHkW4gFfeNcC+94Q4UcIL9EVq0LyUpcxBJcdEm0LkziViyCIjf0Ne4TqDTUFdfKHMQ6WZi3TcilqXBIT77RsR9ZJNGM3VbWgIAABAmSURBVCWdgoNIigu3b0S8lgaP1+zqcKu9dnmOhMpMKUFlJZE0s/XN/Wx/51Do+MCOIwCMnjwI6PqaTfEc9hquT2LqkKldGwarMlNcqKwk0s15Mwl3sDiRNZv83jfihIbBustMKjElhDIHkW4knqu/+pVJxDwM1j1pThPmukyZg0gP5u28PrDjCAd2HAllFhdWW0b27k9p4HoyMomYh8G6J81pwlzCKHMQ6ca8/RPBrCK4Gqw3k4g0simeK766xTQMVhPmukyZg4iERBvp5M4kOrPAXzyGvXrFNAxWI5kSRpmDSA8SKZOI1h/xyfMvULV6NUCoLyLeWQR0IZMA9UF0QqyZg4KDSA/m7sAuq66jqHczpblOQSHSbOt4dlZ7efskvB3YbQKFykydprKSiHSauwO7qbSOKcC8rAEAFNX05YOPqsPOto73aq9u3mGw7mDRruSkMpNvlDmICNC+5HRwbxWf9IFNk/sCycsk3KKWnFRm6pAyBxHpknCd171LalhSEwgOnkzC3R/hZybh1qnOa02YiwtlDiISVkfLdBzJzqDqeCMf9mmm37STQtfdwSJpmUR1KQtqallEtibMeShzEJG46GiZjtE4JaeczN5sClzz7k53Zs4pzMurJytwPVGZxIbaEjZkwJoRE8EMZ4GtIQ47XvdIyhxEJGbeZTq8I53W7a4EYO6EIQDcuPIB8itLyJleAPiTRYCn87r0XaY2NLKi98TWG3pwmUmZg4j4zrtMR8bRJublZ3Plzc7Ma+9WpqtyZzDveCM5B6sYWf4xh6rrmeNDcHCPdFq28jKK2c8yAqWxhmMs2PwEi3pocIiVMgcROWGx7Fa36Df3cnLVAYacdmroXj8yiQ7nS9i+oXMLRp/Hos/8OK7vm6qUOYhIwsWyW92Dm86BTW9RGliuw69Mot18iT9+kzUH3gwdF9s6OPCm+iQ6oMxBROIuUibhLTl9/qn/AOB/lv5r6Fy0rU3jYdkThRTTwNSRZ4bOdXnf6zSgzEFEki5SJuHOIgDeeTmb3rt3cOPKBwCoOt7I2jdm8dIFrSOR/AgWC+wAaDwWmlVdTANUl3bb4BArZQ4i4jtvJuHuj3Av6AdQtbWIkiH5PLHoO0D7kU9xCxTuTYSAZY27KO7TO5RJdLcsQgvviUjK8W5l6t5Twss7ge5QdT1r82by7mnz2gUKiF+wWLni06wxx2DkjNh3q0sDKVVWMsbMBx4BMoFfWWvv91z/FvBloAkoB26y1u71s00iknjuCXXePSWgbSbhXYrjpP27WTywL/9y813t+iu8k++g68FiEdksOrgbbBkrGcKa7P6ha1F3q+uGfMscjDGZwHbgEqAEWA9ca60tct1zAbDOWltrjPlnYJ61dnG0ZytzEElfHS3LMXryoNA5d7DwZhLuYa/eYBFpccCoIuxVHdMeEykqZcpKxpizgXustZcGju8CsNbe18H9ZwA/s9aeE+3ZCg4i3Ue0rUxj2WQolm1PI/LsE7GSGieTGDgSiLLHRIpKpbJSHrDPdVwCzI1w/5eAlzu6aIxZDiwHGDvW3yFuIpI40bYyhUlMWX4f08/LC2URwa1KoW0msXBm63NOqOTk2SdiUeluFo2cAYtWAFH2mOgm/AwOJsy5sGmKMeaLQCHw6Y4eZq19DHgMnMwhHg0UkdQTaRhstKXB3cNkw5WcgvdEVbis7RpMwSXAA9os0xEoOS17pfX+dMgkokl6WckYczHwU+DT1tqyzjxbZSWRniPSMNhYlgaPtkd2RBG2I422rSmkRrBIpbLSemCyMWYCsB9YAlznviHQz/AoML+zgUFEehZ3JuFdliOWTYbcJSeIsewUYTvSSNuaQvqWnXyd52CMWQD8BGco6+PW2nuNMT8ANlhrVxljXgNmAAcDL/nYWvvZaM9V5iDSM0Vb4C/SyCavExrp5M0kIiwFHiw3rZi/IvpzfZRKmQPW2jXAGs+577m+vtjP9xeR7iXaAn/uTKJ2/Xpq169vM/vaHSy8y3gEy07htkFtx51JuLKIjqRjn4TWVhKRtBHLyCbvshzRgoW77BS189rdYe3prPbq1L7XKUjLZ4hI2oo2oS7SGk6R5kzE1HkdobM6HO+EukRlESlVVhIR8VNH+1yDEygO7DjiCh6tWQUQcc5ETJ3XETqrw3FnEqmcRShzEJFuKVpW0VRezohD6xnXtB1wyk4A/WfPBk5gmY5gmSmw9EYkiVyWQ5mDiAiRswqAsiO9Kev7KQ6f6vwm35TXGiy8/RPnAJe5gkXUzuvSzW37IjooM6Vyf4QyBxHpkSJlFk3l5TQdPhy61lJby5g+pZy/wtmtzp1JeJcRv6h2DQsz32bEwCznxXv/5vw97tzWN+8gWPiZSShzEBHphEiZRa9hw+g1bFjo2oEdRzjCRPZ/5RnAWRvo9oIBzP7mwnYlp58ePZc/j1rA88sCJSfPpkKR+iRSKZNQ5iAiEsX6H7/EzqJjoePKTGd11iHNpQBMCgQK6MRIpyRNoEuZJbv9pOAgIsnkDhbeQNHY3EJF3zrWzZwORCk7RSk5xXPYq8pKIiI+m/3NhcwOfO3NKqr7jKavhc9t2A3A5cFggRMcfnjoLH7IWcydMISLTgoEiuCLPSWnZA57VeYgIhJHHZWggkNoy6rrKOrdTGlur3ZZxfcO38Hklj30yTu99YGBTOJEO6uVOYiIJJE7qwD467L/YF/DSOq2OXtSN2WOZAowL2sAF2YNpqKmAXbUA/Bq3W3s6vdXJma8D8D4xl3UVNcxonBZwjurlTmIiPjIu2zHjpI+HBpRSMZAp8O619ChoZFRweG0R7IzAMiv38n0rNfpN/hjAGomX8ncRd8GYh/2qsxBRCSFDF58TZtlw3OCwaKu/azsob2mcGjEbEaHgsVE1jVO5KSG3QxoOcaUI39l6w6n83pm/waqhraOiIp3JqHMQUQkScKtHAutwWJvIFj0GjYslFWc1Nfp6M5qOU5dRj8GjD0DgOLKbRQN+QeZ06tDz3NnEhrKKiKSpiIFC3egADj28fuhAAFwtH4CANVDnSG1tU3HOT7hEP/51e8CCg4iIt2GO1h4s4qq8n1kjKyGKYFlOo6MZPvx8zmWMQBoDRbBUVKfv/1M9TmIiHQH7v4Kb1bRa08p7IH+zQWAEyymjHwqFCweb5lK/6NzKC3tz8jqvHbPjkbBQUQkDXg7tr3Bos/hY2QNK2Dcd53Niwp+8Sn+Muwt6kwWlfs/G/P7qawkItINBDcvypo2zTlRXUrOuOMMnpnDDQ0f8dRXi1VWEhHpaXKuuKLNce22fdRug6rKCXy59GPab4YamYKDiEg3EKnslLuvKebnZcStZSIikjIGL76GcU8/xbinn2Lw0utjfr2Cg4hINzfyu9+N+TUKDiIi0o6Cg4iItKPgICIi7Sg4iIhIOwoOIiLSjoKDiIi0o+AgIiLtKDiIiEg7abnwnjGmGihOdju6KBeoSHYjukhtTw61PTm6W9vHWWuHdfYB6bq2UnEsqwumEmPMBrU98dT25FDbkyMebVdZSURE2lFwEBGRdtI1ODyW7AacALU9OdT25FDbk+OE256WHdIiIuKvdM0cRETERwoOIiLSTloFB2PMfGNMsTFmpzHmzmS3JxpjzB5jzGZjzEZjzIbAuSHGmD8ZY3YE/h6c7HYGGWMeN8aUGWO2uM6Fba9x/Ffge7HJGDMreS3vsO33GGP2Bz7/jcaYBa5rdwXaXmyMuTQ5rQZjzBhjzBvGmA+NMVuNMbcFzqf85x6h7Sn/uQfakmWMeccY80Gg/d8PnJ9gjFkX+OyfN8b0CZzvGzjeGbg+PgXb/oQxZrfrs58ZOB/7z421Ni3+AJnAR8BEoA/wAVCQ7HZFafMeINdz7kHgzsDXdwIPJLudrradD8wCtkRrL7AAeBkwwFnAuhRs+z3A7WHuLQj8/PQFJgR+rjKT1O5RwKzA1wOB7YH2pfznHqHtKf+5B9pjgOzA172BdYHP9AVgSeD8fwP/HPj6/wD/Hfh6CfB8Crb9CeDqMPfH/HOTTpnDHGCntXaXtbYBeA5YmOQ2dcVC4MnA108Cn0tiW9qw1v4VqPSc7qi9C4GnrOMfwCBjzKjEtLS9DtrekYXAc9baemvtbmAnzs9XwllrD1pr3wt8XQ18COSRBp97hLZ3JGU+d4DAZ1gTOOwd+GOBC4HfBc57P/vg9+R3wEXGGJOg5rYRoe0difnnJp2CQx6wz3VcQuQfxFRggT8aY941xiwPnBthrT0Izv9cwPCkta5zOmpvunw/bg2k0Y+7Sngp2fZAmeIMnN8C0+pz97Qd0uRzN8ZkGmM2AmXAn3CymSPW2qbALe42htofuH4UGJrYFrfytt1aG/zs7w189j82xvQNnIv5s0+n4BAuQqf6ONxzrLWzgMuAW4wx5ye7QXGUDt+PXwAnAzOBg8DDgfMp13ZjTDbwe+Ab1tqqSLeGOZdqbU+bz91a22ytnQnk42Qxp4S7LfB3SrXf23ZjzKnAXcA0YDYwBPhO4PaY255OwaEEGOM6zgcOJKktnWKtPRD4uwx4EeeH71AwnQv8XZa8FnZKR+1N+e+HtfZQ4H+gFuCXtJYwUqrtxpjeOP+4/tZa+z+B02nxuYdre7p87m7W2iPAWpx6/CBjTHDdOXcbQ+0PXD+JzpcyfeNq+/xAqc9aa+uBFZzAZ59OwWE9MDkwkqAPTofQqiS3qUPGmAHGmIHBr4HPAFtw2nxD4LYbgJeS08JO66i9q4ClgVEQZwFHg2WQVOGpqV6J8/mD0/YlgdEnE4DJwDuJbh84o0iAXwMfWmt/5LqU8p97R21Ph88dwBgzzBgzKPB1P+BinH6TN4CrA7d5P/vg9+Rq4HUb6O1NtA7avs31C4XB6Stxf/ax/dwkq7e9K39wety349QF7052e6K0dSLOyIwPgK3B9uLUKP8M7Aj8PSTZbXW1+VmcMkAjzm8aX+qovThp6s8D34vNQGEKtv3pQNs2Bf7nGOW6/+5A24uBy5LY7nNx0vtNwMbAnwXp8LlHaHvKf+6BtpwGvB9o5xbge4HzE3GC1k5gJdA3cD4rcLwzcH1iCrb99cBnvwX4Da0jmmL+udHyGSIi0k46lZVERCRBFBxERKQdBQcREWlHwUFERNpRcBARkXZ6Rb9FpOcyxgSHlAKMBJqB8sBxrbX2U0lpmIjPNJRVpJOMMfcANdbah5LdFhG/qawk0kXGmJrA3/OMMX8xxrxgjNlujLnfGPOFwHr7m40xJwfuG2aM+b0xZn3gzznJ/S8Q6ZiCg0h8nA7cBswArgemWGvnAL8Cvha45xHgx9ba2cBVgWsiKUl9DiLxsd4G1qoxxnwE/DFwfjNwQeDri4EC1xYAOcaYgdbZC0EkpSg4iMRHvevrFtdxC63/n2UAZ1trjyeyYSJdobKSSOL8Ebg1eBDc31ckFSk4iCTO14HCwC5dRcBXk90gkY5oKKuIiLSjzEFERNpRcBARkXYUHEREpB0FBxERaUfBQURE2lFwEBGRdhQcRESknf8PNHprv+OO26sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "surv.iloc[:, :5].plot(drawstyle='steps-post')\n",
    "plt.ylabel('S(t | x)')\n",
    "_ = plt.xlabel('Time')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "The `EvalSurv` class contains some useful evaluation criteria for time-to-event prediction.\n",
    "We set `censor_surv = 'km'` to state that we want to use Kaplan-Meier for estimating the censoring distribution.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "ev = EvalSurv(surv, durations_test, events_test, censor_surv='km')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Concordance\n",
    "\n",
    "We start with the event-time concordance by [Antolini et al. 2005](https://onlinelibrary.wiley.com/doi/10.1002/sim.2427)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6556657764212133"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ev.concordance_td('antolini')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Brier Score\n",
    "\n",
    "We can plot the the [IPCW Brier score](https://onlinelibrary.wiley.com/doi/abs/10.1002/%28SICI%291097-0258%2819990915/30%2918%3A17/18%3C2529%3A%3AAID-SIM274%3E3.0.CO%3B2-5) for a given set of times.\n",
    "Here we just use 100 time-points between the min and max duration in the test set.\n",
    "Note that the score becomes unstable for the highest times. It is therefore common to disregard the rightmost part of the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXiU5dn38e85mWQSSCBAEpZA2AVZBdlU3JdqtS5Vq61b3ahtae1eu7f2sa22b/u01T5qq9Uu1gWt0pa6oygqArLJHvawJSFASCDbzPX+MROMcUgmZIZ7Mvl9jmOOzNxL5ry5w5xz7eacQ0REpDmf1wGIiEhyUoIQEZGolCBERCQqJQgREYlKCUJERKLyex1AvOTl5blBgwZ5HYaISIeyePHicudcfrR9KZMgBg0axKJFi7wOQ0SkQzGzLUfapyomERGJSglCRESiUoIQEZGolCBERCQqJQgREYlKCUJERKJSghARkahSZhyESFscqguyq7KGPVW1ZKankZPpp2vAz/5D9WytOMi2ioPsqaojMz2NzHQfWelpZGWk0SXDT9eMNHplBxjYqwuZ6WleXwoAoZAj6BzpafrOJ/GjBCEpJRRyrNpZyZJt+9i+9xDb9x1i575DVNcFqWsIUh90VNbUs+9gfbvfywz6dsuksEcWhtEQChFy0C0rnT7dAvTulklBToD8nAB52eGfhblZ+Jt9iNc2BNm9v5a6YJC6Bkd9MMSh+iAH6xo4WBektj5EMORoCIX3VdU2UFlTz4GaBnbuO8S2vYfYVnEQgJOG9uKskQWcOaKAAT27tPsapXNTgpCU8Mb6Mh5bsJW3N+45/OGfnmb07Z5Fv9xMCnOzyPAbGWk+sjP99O2eRZ9umfTKzqC2IURVTQNVtQ3kZPop6tmFop5d6JUdoK4hRE19kJqGIAfrghysDVJV20DpgRo2lVezubyanftrMIMufj9msO9gHWt3VVJ2oJZQs/W4MtJ8DMnvyvDeOdQ3hFhXeoAtew4SbH5gKzLSfORk+undLZOh+V05c0Q+9UHH6+vK+OFzK4GVTBnck+tPGsh5o/qQ4VfJQtpOCUI6vNU7K7n50UXkZqVzzvG9OWVYLyYP6km/7ln4fNau352VEa5aOhrBkGNPVS1lVbWUV9Wxu7KGDaVVrC+tYum2vaT7fAzvnc2FY/tS1DNcXZWeZqSnhau0ugT8dMlII+D34U/z4fcZaT4jO+BvsWprU3k1L6zcxWMLtjLzsSXk5wS4Zfpgbpo+WFVQ0iaWKkuOTpo0yWkups6npj7IJ37/JvsO1fP87afSKzvgdUhJIxRyvL6+jD/P38y8dWWM6tuNe64Yx5jC7l6HJknEzBY75yZF26evE9Kh/WzOataXVvH/rhyv5NCMz2ecOaKAv9w0hfuvPZGyqlouuW8+v/jvGuoaQl6HJx2AEoR0WC+v2s1f3t7CracO5rTjos5WLBHnj+nDy187nSsm9uf+1zdw1YNvs2PfIa/DkiSnKiZJeg3BEEu37WPeujLW7j7Aofpww/HqHZUM6NmFf37xZAL+5Ohu2hH8Z/lOvjVrGYH0NH539QSmD8/zOiTxUEtVTGqklqQTCjnWlR7gnQ17eHvjHt7asIcDNQ34DIbkZ9M14CfT7+OUYXl8+4KRSg5tdOG4vozsm8Pn/7aY6x5ewNTBPRndrzuj+3Vj6pBeFOZmeR2iJAklCEkqizZXcNvf3qO8qhaAwtwsLhjTh9OPK+CUYb3I7ZLhcYSpYWh+Ns9+8RR++/J63tlUwd/e2UJtQ4j0NOPm6UP40lnD6BrQx0NnpyomSRp7q+u44LdvEEj3MfPMYUwb0kuDvY6RhmCIDWXV/PGNjcxaXELf7pl878LjuWhcP69DkwRTLyZJes45vjlrGRXVddz3mYlcOWmAksMx5E/zMaJPDr+6cjxPf/4kenTJYOZjS3hi4VavQxMPKUFIUnj0rc28vLqUOy4YqX76HjtxYE/+9aXpTBvSkzv/terwNB7S+ShBiOfe376fn81Zw9kjC7jxlEFehyNAms/45RXjAfjmrGWE2jgViKQGJQjxzL6Ddfx8zmou/7+36NE1nV9eOR6z9k2NIfEzoGcXfnDRKN7ZWMGjb2/2OhzxgLopyDFXWlnDk4u28eC8jRyobeCyCYV8/bwR9OyqHkrJ5qrJA3hh5S5+8d81nHZcPkPzs70OSY4hJQg5Jmobgvx72U6eXbqd+cXlhBycNbKAb50/gpF9unkdnhyBmfGLy8dx3m/m8c2nlvHUbSeT1s4JEKXjUIKQhKupD3LrXxbxxvpyBvTM4otnDuOSEwoZVqBvox1B726Z/PjiUXz1iWU88tZmbp4+2OuQ5BhRgpCEqmsI8YW/v8cb68v5+SfHcvXkAWpn6IAuPaGQfy3bya9eWMu5x/emqJe6IHcGaqSWhGkIhvjyP5bw6ppS7rpsDJ+eUqTk0EGZGXddNga/z/j208tJlQG20jIlCEmI8MC35Ty/chc/vGgU10wd6HVI0k59u2fx3QuP5+2Ne/jHu9u8DkeOASUISYgH5m3kn0u28/Vzj+Mm1VmnjKsnD+Dkob348eyVfO+fK9iyp9rrkCSBlCAk7uatK+Oe59dw4bi+zDxrmNfhSByZGb/79AQuP7GQpxaVcOavXuOLj73HW8XlbV5XW5KfJuuTuNqyp5qL751P3+6ZPPOFk+mSoX4Qqaq0soaH5m/isXe2cqC2gYKcAJ8Y349PTRrAiD45XocnMWppsj4lCImL0soa3tu6l9+8tJ5dlTX8a+Z09XTpJA7VBXllzW6eW7qD19aWUh90XDCmD1855zglig5ACwZJwjy7ZDu/enEtJXvDy1d2yUjj/mtPVHLoRLIy0rhoXD8uGtePvdV1/Hn+Jh6ev5nnV+7iE+P6cffl48jK0KJOHZEShBy14tIqvv30co7rncNnTx7ExIE9GN2vm1Z468R6dM3ga+eN4Kbpg3lw3kb+8NoGCnICfP+iUV6HJkchoY3UZna+ma01s2IzuyPK/q+Z2SozW25mr5jZwCb7bjCz9ZHHDYmMU9quIRji608tIysjjYc+O4lbTh3CxKIeSg4CQG6XDL51/kiunVbEQ/M3sXhLhdchyVFIWIIwszTgPuACYBTwaTNr/jViCTDJOTcOmAXcEzm3J/AjYCowBfiRmfVIVKzSdg/M28iybfv46SVjKMjJ9DocSVJ3XHA8/bpn8c1Zy6mpD3odjrRRIksQU4Bi59xG51wd8DhwSdMDnHNznXONq5G8A/SPPP8Y8JJzrsI5txd4CTg/gbFKG6zeWcn/vryOC8f15RPjtSSlHFl2wM/dl49jY1k1v3l5ndfhSBslsg2iEGg63LKEcIngSG4G/tvCuYXNTzCzGcAMgKKiovbEKjEKhRzfeGoZ3bPS+eklY7wORzqA6cPz+PSUIv44byMZaT66ZaYTSPcxqm83Jg3q6XV40oJEJohok+5E7VNrZtcCk4DT23Kuc+5B4EEId3M9ujClLRZurmDljkruuWKc1m+QmH334yNZsnUvv3+1+PA2n8ED103i3FG9PYxMWpLIBFECDGjyuj+wo/lBZnYO8D3gdOdcbZNzz2h27msJiVLa5On3SuiakcZF4/p6HYp0IDmZ6Tz/ldOoD4aobQhRVdPA5/66iC//YwlPfG4a4/rneh2iRJHINoiFwHAzG2xmGcDVwOymB5jZBOAB4GLnXGmTXS8A55lZj0jj9HmRbeKhQ3VB5qzYxQVj+2qEtByV9DQf2QE/fbpn8qcbJtMrO4ObHlnEtoqDrZ8sx1zCEoRzrgGYSfiDfTXwpHNupZndaWYXRw77JZANPGVmS81sduTcCuCnhJPMQuDOyDbx0Asrd1FV28DlE/u3frBIK/JzAjxy42TqGoLc+MhCtu875HVI0oym2pCYXffQAjaWVfPGt87Ep2UnJU7e3rCHGx95F+dgxmlDuO30oXQNqIR6rGiqDWm3nfsP8WZxOV86c5iSg8TVSUN78crXz+Ce59fw+1eLeXzhNj4+pg9ZGX4y033079GFT04o1N+dB5QgJCb/XLId5+CTql6SBCjMzeK3V0/ghpMHcfd/1/DPJdupqQ9RFwwBMHdtKb/+1HiN1D/GlCCkVc45nl5cwqSBPRiU19XrcCSFTSzqwROfO+nw62DI8dCbG/nZnDVUVNXxwPUn0i0zHYCa+iBmdLikcbCugY1l1Wwoq2LNrgOs2VnJhrJqbj97OJefmFxfwJQgpFVLtu1jQ1k1P//kEK9DkU4mzWfMOG0oBTmZfOOpZXzq/rcZ3z+X5dv3s373AUb2zeFfM6d3iLXOH5m/iQfnbWTH/prD2/w+Y1hBNmUHapm/oVwJQjqWUMjxs/+spkeXdC7U2AfxyKUTCumVncHMx5awu3IXYwq7U5hbwMurd7O8ZD/jByT3OIote6q5a85qxhZ259NTihhakM2Q/K4Mycsmw+/j0vvmU3agtvVfdIwpQUiLnly0jUVb9nLPFeMOF+1FvHDq8Hze+8G5+Cy89On+Q/VMuetlnn6vJOkTxD0vrCU9zcf9155IQbePTm6ZnxNg657kGwuiNanliPZU1fLz/65hyqCeXJlkRV/pnNJ8drg6qXtWOueN7sNzS3dQ25C8M8Uu2bqX/yzfya2nDomaHAAKcgKUVSVfCUIJQo7orjmrOVjXwF2XjekQdbzS+VxxYn/2H6rn1dWlrR/sAeccP5uzmrzsADNOO3IbXkFOJhXVddQ1hI5hdK1TgpCo3tpQzjPvbWfGaUMY3lvrCktymj4sj97dAsxaXOJ1KFG9uGo3Czfv5avnDm9x8F9+TgCAPdXJVYpQgpCPCIUcP5m9igE9s/jSWcO9DkfkiNJ8xmUT+vPaurLDjbzBkOOvb2/m5VW78XKmiH0H67j7v2sYmt+VqyYNaPHYgkiCKK1UgpAk9+8VO1m7+wDf/NhIMtM7Vh9z6XyuOLGQYMjx3NLtlB6o4bqHFvCD51Zyy18WccOfF1JcWnXMY1qzq5KL753Ptr0H+fHFo/GntfxRW9AtkiCSrCeTejHJhzQEQ/zvy+s4rnc2F41Vt1ZJfsMKchg/IJc/z9/M/a9vpKq2nrsvH0t1bZDfvLyO8/93HrecOoSvn3cc6a18UMfDv5fv4JtPLScn08/jM07ixIGtr5bcWMWUbF1dVYKQD5m9bAcby6r56jnHae4b6TCumFjI9n2HyO2SzuyZ07lqchE3TR/M3G+cwWUTCrn/9Q1c99AC9iS4p9Af521k5mNLGNWvG//+0vSYkgNAXnZjCaKmlSOPLZUg5LD6YIjfvrKeUX278bHRfbwORyRmV08poltWOueO6v2htUrysgP88srxnDIsj28/vZyL753Pg9efyOh+3eMewxMLt3LXnNVcOLYvv7nqBDL8sX//Tk/z0bNrRtJVMakEIYc9814JW/Yc5GvnqvQgHUt6mo9LTig84kJWl04o5KnbTiLkHJf/31ss2bo3ru///Pu7+M4zKzh1eF6bk0OjgpyAqpgkOdXUB/ndK8WM79+ds48v8Dockbgb1z+X2TOnk5OZzi9fWBu33/v2hj18+fEljB+QywPXnXhUyQHC7RAqQUhS+vHslWzfd4hvnT9Sg+IkZeXnBPjcaUN4a8MeFm5u/yKVi7dUcMujCxnYswt//uzkdi3Fm58ToKwyudoglCCEJxdu4/GF2/jCGUM5ZVie1+GIJNQ1UweSl53B715Z367fs3TbPm54eCEF3TL52y1Tye2S0a7fV5CTSVlVradjN5pTgujk3t++n+8/9z6nDOvF188b4XU4IgmXlZHGjNOG8Mb6chZvObq2iBUl+7nuoQX07JrBY7dOpfcR5lhqi4KcAPVBx76D9e3+XfGiBNGJ7TtYx21/W0yvrhn87uoJpKlhWjqJa6YOpGfXoytFvLWhnGsfWkC3zHQeu3UqfbtnxSWmxrEQydQOoQTRSYVCjq89uYzdlTX84ZqJ9Ir0wxbpDLoG/Nxy6mBeX1fGWxvKWbOrkldW7+bfy3dwqC76zLChkOO+ucVc+6cF5GVn8PiMafTv0SVuMRUk4WA5jYPopB58YyOvrinlJxePZkJRbIN5RFLJ9ScN4sF5G/nMHxd8aHuPLulcd9Igrj9pIHnZASpr6tm+9xC/fGEtr64p5RPj+/GLT45tcfK9o9E4FXgyDZZTguiEFm6u4JcvrOXCsX25/qSBXocj4onsgJ8/XDOR5SX76d8ji8LcLGrqQzw8fxO/e2U997+2gQy/j6raBgDS04w7LxnNddMGJqSnXzJWMSlBdDJ7qmqZ+dh79O+Rxc8vH6surdKpnTw0j5OHfrjn3klDe7GhrIrHFmwlGHL0y82kb/csxhZ2Z1Be14TFkh3w0yUjTVVM4o26hhC3P76UvQfreebzk7WEqMgRDM3P5gcXjTrm75tsg+XUSN1JNARDfPWJpbxZXM7/XDqGMYXxn4tGRNqnICdAaRINllOC6ARCIccdz6zgPyt28v0Lj+dTrSxeIiLeaBwslyyUIFJcfTDET/61klmLS/jKOcO55dQjr4srIt4KT7eRPAlCbRApaE9VLXPe38W8dWW8vWEPVbUN3HrqYG4/W8uHiiSz/JwAB2obOFQXJCvD+9UclSBSzPzicm5/fAnlVXUU5mbxifH9OGtkAeccX6AeSyJJrulguaJe8RuEd7SUIFJE4yjP37y8jiH52Txy4xRG9+umpCDSgXwwFqJGCULio6Y+yBf+/h6vrinl0hP6cddl8R/lKSKJV5ATHk2dLGMh9CnSwTUEQ8x8bAlz15by00tGc22CRnmKSOIVdEuu0dRKEB2Yc47vPLOCl1fv5qeXjOa6kwZ5HZKItEPPLhmk+Sxp5mNKaDdXMzvfzNaaWbGZ3RFl/2lm9p6ZNZjZFc32Bc1saeQxO5FxdlR3P7+WpxaXcPvZw5UcRFKAz2fkZWekfhWTmaUB9wHnAiXAQjOb7Zxb1eSwrcBngW9E+RWHnHMnJCq+ju6pRdu4//UNXDutiK+co+6rIqmiICezU1QxTQGKnXMbAczsceAS4HCCcM5tjuwLJTCOlFN2oJaf/nsVUwb35CcXj1Gbg0gKyc8JsGt/uIqpIRiiui5I96zY5k1rCIZYVrKfN9aXUZCTyWemFrUrlkQmiEJgW5PXJcDUNpyfaWaLgAbgF865Z5sfYGYzgBkARUXt+4foSO76zyoO1Qf52WVjtQqcSIopyAkwb10Zk/7nJfZU1+EcfGx0b75/4SgG9Pxw19dQyLGu9AALNlbw9oY9vLWhnMqa8PTkZjBxYC4j+3Q76lgSmSCifXK1ZTXuIufcDjMbArxqZiuccxs+9MucexB4EGDSpEnJs9J3Ar2xvoxnl+7gy2cPZ1hBttfhiEicXTqhkIrqOnplB8jPCVDXEOLRtzZzztrXue30oUwb0ov3tu5lyda9LN6yl72RNawLc7O4YExfTj0uj9H9unPxvW/yy+fX8tBnJx91LK0mCDPrAnyd8Af2rWY2HBjhnPt3K6eWAE1nhesP7Ig1MOfcjsjPjWb2GjAB2NDiSSmupj7I9599n8F5XfnCGUO9DkdEEmDakF5MG9LrQ9uuP2kgP5uzmt++sp7fRtbRHpLflbOP783UwT2ZNqTXR0oXXzhjGHc/v4YFG/cwtdnvi1UsJYg/A4uBkyKvS4CngNYSxEJguJkNBrYDVwOfiSUoM+sBHHTO1ZpZHnAKcE8s56ay372yni17DvLYLVPJTPd+nhYROTb65WZx72cmcsup+9hbXccJA3Lp0TWjxXNuPGUQj761mV88v4ZnPn/yUbVVxtLNdahz7h6gHsA5d4jo1Ucf4pxrAGYCLwCrgSedcyvN7E4zuxjAzCabWQlwJfCAma2MnH48sMjMlgFzCbdBrProu3Qezy7Zzh9e28CnJvXn5GF5rZ8gIinnhAG5nDmyoNXkAJCZnsZXzx3Okq37eGHl7qN6v1hKEHVmlkWk/cDMhgIx9cFyzs0B5jTb9sMmzxcSrnpqft5bwNhY3qMzmLeujG88tYxpQ3ry00vHeB2OiHQQl0/szx/f2MQ9z6/hUH3D4e2nDM2joFtmq+fHkiB+BDwPDDCzvxOu7vns0YUrbbW8ZB+3/W0xw3vn8OD1kwj4VbUkIrHxp/n43seP55a/LOKrTyw7vP0zU4v42WWtfwdvMUFYuNJqDfBJYBrhqqXbnXPl7YpaYrJ21wFuemQhPbpk8OiNWkNaRNruzJEFvPOds6muDZcgrn1oAQdqGlo5K6zFBOGcc2b2rHPuROA/7Y5UYrZ4SwU3PbKIzHQff7l5SkzFQRGRaPJzAoenEs8O+KmtD8Z0XiyN1O+Y2dF3pJU2m7u2lGv+tIAeXdKZddvJDM3XeAcRiY9Aehq1DbFNXhFLG8SZwOfMbAtQTbiayTnnxh19iHIkc9eUcutfFjGiTw6P3jSFvOyA1yGJSAoJ+H3UxFiCiCVBXNC+cKQt7n99A/17ZPH4jGnkqM1BROIs4PfF3AbRahWTc24LkAt8IvLIjWyTOKuormPh5gouHt9PyUFEEiKzDVVMrSYIM7sd+DtQEHn8zcy+1K4IJapXVu8m5OC80X28DkVEUlTA76O2IX5VTDcDU51z1QBmdjfwNvD7o45Qonpx1W4Kc7MY3e/oZ18UEWlJwJ9GbX2cShCEG6WbppsgMUy1IW1zsK6BeevKOHdUb63vICIJE0iPbwniz8ACM/tn5PWlwENHGZscwbx15dQ2hDhvdG+vQxGRFJbZhhJEqwnCOffryHTb0wmXHG50zi1pV4TyES+u2kX3rHSmDOrpdSgiksLCJYg4JQgzmwasdM69F3mdY2ZTnXML2hemNGoIhnhldSlnH1+APy2WWj8RkaMT8PuoC4YIhlyrK1LG8mn0f0BVk9fVkW0SJ+9uqmD/oXrOG6XeSyKSWI0TftbFUIqIqZHaOXd4OU/nXIjELlXa6by4ajeZ6T5OPy7f61BEJMUF/OGP/VgaqmNJEBvN7Mtmlh553A5sbF+I0sg5x0urdnPq8HyyMjSVt4gkVuNqlLG0Q8SSIG4DTia8bGgJMBWYcfThSVNrdx9g+75DnHN8gdehiEgn0FiCiGU+plh6MZUSXk9aEuDVNaUAnDFCCUJEEi+Q3ljFFIcShJndY2bdItVLr5hZuZld2/4wBeC1NWWM7teN3lrvQUSOgcZG6ljGQsRSxXSec64SuIhwFdNxwDfbEZ9E7D9Yz+KtezlrpEoPInJsZKbHt5G6cVrRjwP/cM5VHHVk8iHz1pcRDDlVL4nIMdNYgqiJoQQRS3fVf5nZGuAQ8AUzywdq2hOghM1dW0qPLumcMCDX61BEpJOIazdX59wdwEnAJOdcPXAQuKR9IUoo5Hh9bRmnH5ff6mhGEZF4aUsjdUwD3pxze5s8ryY8mlraYfn2/eypruNMtT+IyDGU2dhIHac2CEmAuWtK8RmcNlyjp0Xk2GksQcTSBtFigrCwAfEJS5qau7aUCUU96NE1w+tQRKQT+aCbaztLEJE5mJ6NS1RyWNmBWpaX7Ff3VhE55j5opI7POIh3zGxyO2OSJmYtLgHgTHVvFZFjrC0JIpZG6jOB28xsM+HGaSNcuBh39CF2XuVVtfxhbjFnjyxglNaeFpFjzJ/mw++z+MzFBFzQ/pCk0W9eWseh+iDfvfB4r0MRkU4q4I9tVblYxkFsAQYAZ0WeH4zlPPmotbsO8I93t3LttIEMzc/2OhwR6aQC6Wnx6eZqZj8Cvg18J7IpHfhbu6LrhJxz/M9/VpGTmc7tZw/3OhwR6cQy/b64TdZ3GXAxkcFxzrkdQE67ouuEXl1Tyhvry/ny2cPVtVVEPBUuQcQnQdRFurs6ADPrGmsQZna+ma01s2IzuyPK/tPM7D0zazCzK5rtu8HM1kceN8T6nslmy55qvvnUMmb8dTFD8rty3bSBXockIp1cwO+LWyP1k2b2AJBrZrcCNwF/bO0kM0sD7gPOJTxN+EIzm+2cW9XksK3AZ4FvNDu3J/AjYBLhxLQ4cu5eOohQyPGj2St57N2tpPmM66YN5AtnDiXDr+YbEfFWrI3Usawo9yszOxeoBEYAP3TOvRRDDFOAYufcRgAze5zwJH+HE4RzbnNkX/NIPwa81Di1uJm9BJwP/COG900KS7bt46/vbOHyif351vkjtCCQiCSNWBupY52s7yUglqTQVCGwrcnrxvWsj/bcwuYHmdkMIutjFxUVtTG8xJpfXI4ZfO/C4+mpNgcRSSIBv4+q2oZWjztifYeZvRn5ecDMKps8DphZZQwxRJvD2sVwXsznOucedM5Ncs5Nys9Prknv3iwuZ3S/bkoOIpJ0Av609k3W55ybHvmZ45zr1uSR45yLZQhwCeHxE436AztiOK+953quuraBJVv3Mn1YciUtEREIz+ja7nEQZuYzs/ePMoaFwHAzG2xmGcDVwOwYz30BOM/MephZD+C8yLYO4d1NFdQHHdOH5XkdiojIRwTiMQ7CORcClplZmyv4nXMNwEzCH+yrgSedcyvN7E4zuxjAzCabWQlwJfCAma2MnFsB/JRwklkI3NmR1sJ+s7icDL+PSYN6eB2KiMhHZMY4DiKWRuq+wEoze5cmK8k55y5u7UTn3BxgTrNtP2zyfCHh6qNo5z4MPBxDfElnfnE5kwf1IDM9zetQREQ+IlyCiE8vpp+0P5zOo/RADWt2HeDb54/0OhQRkagC/jiVIJxzrzc+N7M8YE9kZLVE8VbxHgC1P4hI0gr4fdQFQ4RCLX+Ut9TNdZqZvWZmz5jZhEhj9fvAbjM7P87xpow3i8vJ7ZKutR5EJGk1Vn/XBVsuRbRUgrgX+C7QHXgVuMA5946ZjSQ8ovn5uESaQpxzvLm+nFOG5pHmizaUQ0TEe42ryrU2H1NLvZj8zrkXnXNPAbucc+8AOOfWxCvIVLOhrJpdlTWcouolEUligfTYlh1tKUE0PfNQs31qg4jixVW7ALU/iEhyC/jDVUytjYVoqYppfGRKDQOymkyvYYBmnmtmW8VB7n21mFOH51HUq4vX4YiIHFHm4RJEy1VMR0wQzjl14o9RKOT41qzl+Mz4xeXjvA5HRKRFjSWI1uZj0uIEcfDYu+0nvp8AAA4USURBVFt5e+Mevvvx4ynMzfI6HBGRFjU2UrdWglCCaKdtFQf5+ZzVTB+Wx6enDGj9BBERj32QIFSCSJjahiDfeGoZAD//5FjM1LVVRJJf4ziIo26DkJYFQ46vPbmMBZsq+PWnxjOgpxqmRaRjaOzmqjaIBHDO8YPn3uc/y3fynQtG8smJUecbFBFJSoe7uaoNIv7+34vreGzBVm47fSifO32o1+GIiLTJ4TYIlSDi6/V1Zdw7t5irJw/g2+eP8DocEZE2+6ANQgkibpxz/P6V9fTrnsmdl4xRo7SIdEjxmItJmnl3UwWLtuzlc6cPJcOvfzoR6ZjUzTUB7nttA3nZGVw1WeMdRKTj8qf5SPOZGqnjZUXJfuatK+Om6YO1lKiIdHiZfp8aqePlvrnF5GT6uW7aQK9DERFpt0B6GjUqQbTf+t0HeH7lLj578iByMtO9DkdEpN0CKkHEx6NvbyYz3ceNpwz2OhQRkbgI+H1qpI6HxVv2MWVwL3p2zfA6FBGRuMhMT1MjdXsdqguybvcBxhV29zoUEZG4UQkiDlbtrCQYcoztrwQhIqkj4E/TQLn2WlGyD4BxShAikkIC6SpBtNvy7fvJyw7Qp5uW4RaR1KFeTHGwomQ/4/p317xLIpJSAmqkbp/q2gaKy6oYqwZqEUkxAb9PCwa1x8odlTin9gcRST0Bf5raINpjeaSBWj2YRCTVhLu5qorpqK3Yvp++3TMpyFEDtYiklvBAOZUgjtqKkv1qfxCRlBTw+6jzMkGY2flmttbMis3sjij7A2b2RGT/AjMbFNk+yMwOmdnSyOP+RMYZTWVNPRvLq9X+ICIpKZDe+se/P1FvbmZpwH3AuUAJsNDMZjvnVjU57GZgr3NumJldDdwNXBXZt8E5d0Ki4mvN+9v3AzC2f65XIYiIJEzA3/q6NoksQUwBip1zG51zdcDjwCXNjrkEeDTyfBZwtiXJgIMVJZEEoSomEUlBmTGUIBKZIAqBbU1el0S2RT3GOdcA7Ad6RfYNNrMlZva6mZ2awDijWr59P/17ZGkGVxFJSbGUIBJWxQREKwm4GI/ZCRQ55/aY2YnAs2Y22jlX+aGTzWYAMwCKioriEPIHGkdQi4ikooDf2xJECTCgyev+wI4jHWNmfqA7UOGcq3XO7QFwzi0GNgDHNX8D59yDzrlJzrlJ+fn5cQt8d2UNWysOMl7tDyKSorxOEAuB4WY22MwygKuB2c2OmQ3cEHl+BfCqc86ZWX6kkRszGwIMBzYmMNYPmbumFIDTR8Qv6YiIJJPMdA+rmJxzDWY2E3gBSAMeds6tNLM7gUXOudnAQ8BfzawYqCCcRABOA+40swYgCNzmnKtIVKzNvbqmlH7dMxnRO+dYvaWIyDEVSwkikW0QOOfmAHOabfthk+c1wJVRznsaeDqRsR1JbUOQN4vLuWxCoWZwFZGUFYihBKGR1M0s3LSXg3VBzhpZ4HUoIiIJ43UbRIf06ppSMvw+Th6a53UoIiIJE0sbhBJEM3PXlnLSkF5kZbT+jyci0lGpBNFGm8qr2VRereolEUl5ShBt1Ni9VQlCRFKdGqnbaO7aUoYVZDOgZxevQxERSahMlSBiV13bwIKNFSo9iEin4E/zkeZruSu/EkTE/OJy6oIhztDoaRHpJFprh1CCiFi0ZS8Zfh8nDuzhdSgiIseEEkSMlm7dx+h+3WKaAldEJBW0NhZCCQJoCIZYvn0fJwzQ7K0i0nmoBBGDNbsOUFMfUoIQkU6ltRoTJQhg6bZ9AEwsUvuDiHQegVaWHVWCIJwgenXNoH+PLK9DERE5ZlTFFIOl28LtD5reW0Q6EzVSt2L/oXqKS6vU/iAinY5KEK1YXhJufzihSAlCRDoXNVK3YunWfZjBeJUgRKSTUQmiFUu37WNofjbdMtO9DkVE5JhqbUbXTp0gnHMs2aYBciLSOakE0YJtFYeoqK5TghCRTknjIFqwZNteACUIEemU1EjdgqXb9pGZ7mNknxyvQxEROeY+M6Woxf2dNkEsL9nH7KU7OGFALv60TvvPICKdWJ/umS3u75SfjC+u3MWnHnibzPQ0/ufSMV6HIyKSlPxeB3AslVfVMmtxCXc/v4Zxhd350w2Tyc8JeB2WiEhSSvkEsWpHJfe8sIZVOyopPVALwMdG9+Z/r5pAVoYWBxIROZKUTxB3/nslq3ZUcs6o3ozq240xhd2ZMqgnvlYW6xYR6exSOkFs2VPNOxsr+Pq5x/Gls4d7HY6ISIeS0o3UsxaXYAZXTOrvdSgiIh1OyiaIYMgxa3EJpw3Pp293LQQkItJWKZsg3lhfxs79NXxq0gCvQxER6ZBSNkE8taiEHl3SOWdUgdehiIh0SCmZICqq63hx1S4unVDY6lwjIiISXUIThJmdb2ZrzazYzO6Isj9gZk9E9i8ws0FN9n0nsn2tmX2sLe/73NLt1AcdV01W9ZKIyNFKWDdXM0sD7gPOBUqAhWY22zm3qslhNwN7nXPDzOxq4G7gKjMbBVwNjAb6AS+b2XHOueCR3m/d7gOc8+vXAdi1v4Zx/bszsk+3hFybiEhnkMhxEFOAYufcRgAzexy4BGiaIC4Bfhx5Pgu418wssv1x51wtsMnMiiO/7+0jvVlmehojeodnZR3RJ4drprY8S6GIiLQskQmiENjW5HUJMPVIxzjnGsxsP9Arsv2dZucWNn8DM5sBzAAoKirivmsmxi14EZHOLpFtENHmsnAxHhPLuTjnHnTOTXLOTcrPzz+KEEVE5EgSmSBKgKatxP2BHUc6xsz8QHegIsZzRUQkgRKZIBYCw81ssJllEG50nt3smNnADZHnVwCvOudcZPvVkV5Og4HhwLsJjFVERJpJWBtEpE1hJvACkAY87JxbaWZ3Aoucc7OBh4C/RhqhKwgnESLHPUm4QbsB+GJLPZhERCT+LPyFveObNGmSW7RokddhiIh0KGa22Dk3Kdq+lBxJLSIi7acEISIiUSlBiIhIVCnTBmFmB4C1XscRJ3lAuddBxEmqXEuqXAfoWpKRl9cx0DkXdSBZKi05uvZIDS0djZkt0rUkl1S5DtC1JKNkvQ5VMYmISFRKECIiElUqJYgHvQ4gjnQtySdVrgN0LckoKa8jZRqpRUQkvlKpBCEiInGkBCEiIlGlRIJobe3rZGdmm81shZktNbNFkW09zewlM1sf+dnD6zibM7OHzazUzN5vsi1q3Bb2u8g9Wm5mSbW60xGu5cdmtj1yX5aa2ceb7DvqNdMTycwGmNlcM1ttZivN7PbI9g53X1q4lo54XzLN7F0zWxa5lp9Etg82swWR+/JEZOZrIjNZPxG5lgVmNsiTwJ1zHfpBeKbYDcAQIANYBozyOq42XsNmIK/ZtnuAOyLP7wDu9jrOKHGfBkwE3m8tbuDjwH8JLwY1DVjgdfwxXMuPgW9EOXZU5O8sAAyO/P2leX0Nkdj6AhMjz3OAdZF4O9x9aeFaOuJ9MSA78jwdWBD5934SuDqy/X7g85HnXwDujzy/GnjCi7hToQRxeO1r51wd0Lj2dUd3CfBo5PmjwKUexhKVc24e4WnamzpS3JcAf3Fh7wC5Ztb32ETauiNcy5EcXjPdObcJaFwz3XPOuZ3Oufcizw8Aqwkv19vh7ksL13IkyXxfnHOuKvIyPfJwwFnArMj25vel8X7NAs42s2grbSZUKiSIaGtft/RHlIwc8KKZLY6ssw3Q2zm3E8L/UYACz6JrmyPF3VHv08xI1cvDTar5OsS1RKolJhD+ttqh70uza4EOeF/MLM3MlgKlwEuESzj7nHMNkUOaxnv4WiL79wO9jm3EqZEgYlq/Osmd4pybCFwAfNHMTvM6oAToiPfp/4ChwAnATuD/RbYn/bWYWTbwNPAV51xlS4dG2Zbs19Ih74tzLuicO4HwEspTgOOjHRb5mRTXkgoJosOvX+2c2xH5WQr8k/Afz+7Gon7kZ6l3EbbJkeLucPfJObc78p86BPyRD6orkvpazCyd8Afq351zz0Q2d8j7Eu1aOup9aeSc2we8RrgNItfMGufEaxrv4WuJ7O9O7FWgcZMKCSKWta+Tlpl1NbOcxufAecD7fHi97huA57yJsM2OFPds4PpIr5lpwP7GKo9k1awu/jLC9wWSeM30SD31Q8Bq59yvm+zqcPflSNfSQe9LvpnlRp5nAecQblOZC1wROaz5fWm8X1cAr7pIi/Ux5XXrfjwehHtirCNcp/c9r+NpY+xDCPe8WAasbIyfcH3jK8D6yM+eXscaJfZ/EC7i1xP+xnPzkeImXGS+L3KPVgCTvI4/hmv5ayTW5YT/w/Ztcvz3IteyFrjA6/ibxDWdcFXEcmBp5PHxjnhfWriWjnhfxgFLIjG/D/wwsn0I4SRWDDwFBCLbMyOviyP7h3gRt6baEBGRqFKhiklERBJACUJERKJSghARkaiUIEREJColCBERicrf+iEi0pSZNXYZBegDBIGyyOuDzrmTPQlMJM7UzVWkHczsx0CVc+5XXsciEm+qYhKJIzOrivw8w8xeN7MnzWydmf3CzK6JrAmwwsyGRo7LN7OnzWxh5HGKt1cg8gElCJHEGQ/cDowFrgOOc85NAf4EfClyzG+B3zjnJgOXR/aJJAW1QYgkzkIXmdfIzDYAL0a2rwDOjDw/BxjVZKr/bmaW48LrH4h4SglCJHFqmzwPNXkd4oP/ez7gJOfcoWMZmEgsVMUk4q0XgZmNL8zsBA9jEfkQJQgRb30ZmBRZHW0VcJvXAYk0UjdXERGJSiUIERGJSglCRESiUoIQEZGolCBERCQqJQgREYlKCUJERKJSghARkaj+P2lYEiCnSrP6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "time_grid = np.linspace(durations_test.min(), durations_test.max(), 100)\n",
    "ev.brier_score(time_grid).plot()\n",
    "plt.ylabel('Brier score')\n",
    "_ = plt.xlabel('Time')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Negative binomial log-likelihood\n",
    "\n",
    "In a similar manner, we can plot the the [IPCW negative binomial log-likelihood](https://onlinelibrary.wiley.com/doi/abs/10.1002/%28SICI%291097-0258%2819990915/30%2918%3A17/18%3C2529%3A%3AAID-SIM274%3E3.0.CO%3B2-5)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8ddnJpNMIAlbQgj7LgYRgYiIS7XVKq7XrVdcqq29tLW2drn9Vdtba72997a12k2udb21m7hbbam4FhUUAWULCIaAEkLIAmQj28x8f39kwBADBJLJmcm8n49HHpmzzMzn5CTzyXc35xwiIpK8fF4HICIi3lIiEBFJckoEIiJJTolARCTJKRGIiCS5FK8DOFLZ2dlu9OjRXochIpJQVq5cWemcy+noWMIlgtGjR7NixQqvwxARSShm9uHBjqlqSEQkySkRiIgkOSUCEZEkp0QgIpLklAhERJKcEoGISJJTIhARSXIJN45ApLOaQxGq6puoqG2ioTlMWsBPMODDb0ZFXRNl1Y3sqG6kORQhNcVHis8I+H0Eo+cFA36yM9IYkhVkcFYawYB//2s75zCzHr8mr95XejclAkk4tY0trC+tYV1pDR9V1VNe28TOmkZ2722hqSVMUyhCcyhCbVOoW9831e8j7BzhiMMM+qUHGNg3lYF9UunfJ0BWeoB+6QGyM9IYPiCd4QP6MKRfkKq6JrbvbqBkdwO1TSGcc0ScIxyBlnBk/1ck8vF7hSKOxlCYppYIDS0hdtW3UFnXxK76ZoZkBTlx9AAKRg9k1thBjMvpq+QgXaJEIAmjqLyOm/7yLu+X1e7flxlMITcryODMNCYPTScY8JOa4iPV72Ng31RyMtPIzkijT6qfpugHa0vEkZ2RSl6/dIZkBUlL8dESiRAKO5pDEZpCERpbwuxtDlNZ10RZTSM7qxupbw7j94HfDAfs2dvCrr3N7KprpnRPIxt21FLd0ELdYRKQz8Bnhs+M1BQfAb+R4m8tqezj9xlpAR9pKX7SAz6G9U9n6vB+9O+Tyke76nmzqIpnV5UCMDa7L2dPzuWz+UOYPrK/koIcMSUCSQilexr4/EPLaA5H+M7ZEzlueD+OG9qPnMy0bnn9NJ+ftBTo2w0vt7c5RMnuBrbt2svOmiayM1IZNiCd4f370K9PoOtvQGsV0YdVe3n9gwpeWr+Th97Ywn2LizlhRH++ffZETpuQrYQgnWaJtlRlQUGB01xDyWVXfTNX/G4p5TVNPDpvFscN6+d1SHGnuqGFv60p5X9f28z2PQ0UjBrA/zt3EjPHDPQ6NIkTZrbSOVfQ4TElAok3LeEIdY0h9raEqW8K8d0nVvN+WS1/+OJMTho7yOvw4lpTKMzjK0qY/2oRZTWNXDJtGLeeN4nBmUGvQxOPKRFIXNtd38yqbXtYvnUXK7buZlXJHppDH7ec+n3GfdfM4Kz8XA+jTCwNzWHmv1bE/a8Xk5bi46ZPj+eU8dmMH5xxQO8nSR5KBBJ3lhVX8eg7H7Fq2x62Vu0FIMVnTB7Wj4JRAxg+IJ0+qX7SU1OYMDiDY/OyPI44MRVX1PGj5wp544NKoLWhenR2X847Lo8vnDKaQRnd08Yi8U+JQOLK8q27uObBZWSkpVAwegBTR/TnhOhXn1T1X+huzjm2VNbzflktG8tqWV2yh8WbKkhL8TF35ki+fPo4hvRT1VFvp0QgcWPTzlouv3cp2ZlpPPmV2Qzsm+p1SEmpqLyWe/9ZzLOrttMvPcAzN85m1KC+XoclMXSoRKApJqTHlO5p4LqH3yEY8PPIF2YqCXho/OBM7vrcVBZ98zQizvGF3y+nem+L12GJR5QIJOaaQxGeX13KNQ8uo64xxO+/MJMRA/t4HZbQmhDuv7aAkl0NfOVPKw9opJfkoUQgMeGc44OdtfzshfeZ/dNX+Pqj79ESifDAdQXkD1XDbzyZOWYgP7t8Cm8VV/GDZ9aSaNXF0nVqmZNuVVRex9PvlvBCYRnFFfX4DD5zbC5XnzSS0yfk4PNptGs8umTacLZU7uU3r3zAmJy+3HjGeK9Dkh6kRCDd5vnVpXznidWEI46Txw7iC6eM4bP5ueRmqUdKIvjWWRPYWlnPz1/YyNjsvpx7XJ7XIUkPUSKQLnPOcc+rRdz10iZOHD2A+VdP10jWBGRm/Pzy49m2ey/ffGwVT/Tvw5Thms4jGaiNQLqkORThO4+v5q6XNnHptGH86UsnKQkksGDAz/3XFjCobxo3PLKc7XsavA5JekBME4GZnWtmG82syMxuOcg5nzOz9WZWaGZ/iWU80r1awhG+8eh7PP3edr599kTu+txU0lI0fUGiy8lM46HrC6hvCnHmL/7JzQve463NVWpE7sViNqDMzPzAJuBsoARYDsx1zq1vc84E4HHg08653WY22DlXfqjX1YCy+BAKR7j5sVX8fc0Obrsgny+eOsbrkKSbFZXX8Ye3tvLMe9upbQwxcmAfPpufy9n5ucwYNYAUvyoUEoknI4vN7GTgdufcOdHtWwGcc//T5pyfA5uccw929nWVCLwXjji+8/gqnl1VyvfPm8S808d5HZLEUGNLmH+s28FfV5WytKiK5nCEAX0CXDZ9ONefMprhAzQmJBEcKhHEsrF4GLCtzXYJcFK7cyYCmNkSwE9r4nghhjHJUdq+p4ElH1Sy4sNdLN+6my2V9Xz3nGOUBJJAMODnkmnDuWTacOqaQryxqYK/rd3B/y3dysNLtjDnuDxu+vR4TQyYwGKZCDrqMN6++JECTADOAIYDb5jZcc65PQe8kNk8YB7AyJEjuz9SOaT1pTVceu8SGlsi9O8TYMbIAdx4xjiuKBjhdWjSwzLSUpgzJY85U/Io3dPAI29t5dFlH/H6pgqe+dpsxg/O9DpEOQqxTAQlQNtPiuFAaQfnvO2cawG2mNlGWhPD8rYnOefuB+6H1qqhmEUsn1Dd0MJX/7ySfukBnv3aTI7JzdQSiALA0P7p3DrnWD5/8mguvudNbnhkBc/eeAoDNIdUwolla89yYIKZjTGzVOBK4Ll25zwLnAlgZtm0VhUVxzAmOQKRaFvA9t0N/O/V05k0JEtJQD5hWP907rt2Bjv2NPK1v7xLS1jzFSWamCUC51wIuAlYBGwAHnfOFZrZHWZ2UfS0RUCVma0HXgO+65yrilVMcmTuXbyZlzeU84Pzj2XGKK19Kwc3Y9RA/vvSKSzdXMV/PLOOtSXVbNu1l9pGzWiaCLQegXRo5Ye7ueJ3S7ng+KH8+soTVBKQTvmfhRu47/UDC/WXTBvGzy47ntQUdTf1kle9hiSB3fPqBwzsm8r/XDpFSUA67ZY5kzhvSh7ltU3s2dvM+h01/N+Sreze28y9V88gPVUDDuOREoF8QlF5La9trOBbZ02kb5p+RaTzzIypI/ofsG9ibibff2Yt1z60jIeuP5F+6QGPopODUVlNPuGhN7eQluLjmlnqqitdN3fmSO6ZO53VJXu4ZP4Snl9dSjiSWFXSvZ0SgRygsq6Jp97dzqXThzMoI83rcKSXOP/4PB75wkzM4OuPvsfZdy/m8eXbWLe9mu17GtjbHNJcRh5SuV8O8Ke3P6Q5FOEGzR0k3Wz2+Gxe+taneKGwjHteLeL/PbXmgOOzxg7k11dO67XrV7SEI9Q2huJyrW4lAtmvsSXMH9/6kE9PGsz4wRlehyO9kM9nnDcljznHDWF1STVl1Y3s2dtM6Z4GHnhjC+f/5g1+M3cas8dlex3qUQuFI2zb3UBxRR3FFfVs3FnL+tIaisrraIlEWPzvZzJyUHzNz6REIPs98952quqb+dJpKg1IbJkZJ4zof8DcAxdMHcpX/rSSax5cxudPHk1LOMKWynpKdjfw/fMmJcSKaeU1jVx0zxLKahr378vOSOXYvCwm5g7h2VWlbK6oUyKQ+FRZ18SvXt7EccOyOHnsIK/DkSQ0MTeT5246lVufXsvvl26lX3qAsTl9aQqFufulTZwzeUjcd2X+5cubqKpv4n8uncLE3EzGZvfdP+VG6Z4Gnl1Vyo7qxsO8Ss9TIhDCEcc3F6xi994WHr7+xLj/Y5PeKyMthd/OncbPLptCesCPmfHUyhK+88RqFm+q4IxjBnsd4kFt2lnLY8u3cd3s0cyd+ckedzmZafiMA0oL8UK9hoTfvvoBbxZVcsdFk5k8VGvUivf6pKbs/4fkwqlDyc1K48E3tngc1aH99B/v0zcthW98ekKHxwN+HzmZaZRVx9/yn0oESe7NDyr59SsfcOm0YfzriZpWWuJPaoqP62eP4c2iStaX1ngdToeWFlXy6vvl3HTm+EPOvjokKxiXVUNKBElsV30zNy94j/E5GfzkkuNUJSRx66qZI+mT6ufBNz+ex6glHGFtSTUhj2c7jUQc/7VwA8P6p3Pd7NGHPHdIvyA7VTUk8eSuFzeyp6GFe66aTp9UNRdJ/OrXJ8DnCkbw3KpSyqobeWXDTs751etceM+bfObuxTy5ssSThOCc4zevfkBhaQ3fPecYgoFDz6WU1y9dJQKJHxt21PDoOx9x7axRHDNEq0pJ/Lvh1DFEnOPC6CI4OPjhBflkpKXw70+s5qy7F/PaxvIei6euKcSNf36XX738ARdNHcpFU4ce9jm5WUFqG0PUN4V6IMLOUyJIQs457nh+PVnpAb55VscNWyLxZsTAPvzLtGG0hCPcfmE+i751OjecOoa/ff1U7rt2BqkpPr74++XMf60o5tNVFFfUccn8JSwqLOP7503i11eegM93+KrVvH6to6bjreeQ6gOS0KLCnbxVXMV/XjyZ/n3ib7i7yMHceflUnHOk+D/+H9bMOGfyED41MYfvPbWGOxdtZH1pDXdecXxMqjwLS6u55sFlAPzxhpM4ZXznR0EP2ZcIqhsZlxM/o/eVCJJMY0uY/1q4nmNyMzvs6ywSz/w+Azr+zzsY8POrfz2ByUOz+Ok/3uejXXt58qsnk5bSfWsgrNtezTUPLaNPwM+j82YxalDfI3r+kOg8SvHWTqCqoSTzy5c2sW1XA7ddmH/Af1UivYGZMe/0cdxz1XTWbq/moTe7b+zBuu3VXP3gMvqmprBg3slHnATg4xJBvPUc0idBEnlk6Vbue72Yq04aeUTFWZFEc96UPD6bn8tvXyliRzcM4Fq1bQ9XPfA2GWkpLJg366jnCgoG/AzoE+iWmLqTEkGSWLh2B7c/X8jZ+bnccdFkr8MRibkfXpBPxDl+8vcNXXqdpUWVXP3A2/Tvk8qCebMYMbBrE8blZgUpU9WQ9LS3i6v45oJVTB85gN/OnaYqIUkKIwb24cYzxvP3NTtYWlR5VK/xYmEZ1/9+OcMGpPPkV07uchKA1p5DaiOQHlVUXse8P6xg5KA+PHRdwWEHvIj0Jl/+1FhGDEznR88VsnrbHpYVV7F4UwXbdu095PMiEccjS7fy1T+/y7F5WTw272QGd9OCOUP6pcddG4F6DfViu+qb+eLvl5Oa4uP/rj9RXUUl6QQDfm67YDL/9ocVXDx/yf79fp9x1cyRfOMzE8jJPHBJ1qLyWm59ei3Lt+7mjGNyuOeq6WSkdd9H5ZCsIJV1zTSFwt3ao6krYpoIzOxc4NeAH3jQOffTdsevB+4Etkd33eOcezCWMSWLplCYL/9xBTtrGnm0G+o1RRLV2fm5PH3jbHbXNxMM+ElN8fH86lL+suwjnn63hKtOGkn/Pqm0hCPsqm9mwTvbSE/184srpnLZ9GHdPgfXvkFl5TVNcfN3GbNEYGZ+YD5wNlACLDez55xz69ud+phz7qZYxZGMnHPc8lTrfzT3XDWN6SMHeB2SiKfa/w2cOHog188ezZ2LNvJAm+mtU6JLad52YT7ZGWntX6ZbDGkzurjXJwJgJlDknCsGMLMFwMVA+0Qg3SgScfz4+UKeeW873zl7Ihccf/j5T0SS0dicDO69ZgaNLWHMIODzdWqaiK7alwjiqcE4lo3Fw4BtbbZLovvau8zM1pjZk2bW4YT4ZjbPzFaY2YqKiopYxNorhCOO7z21hkfe+pB5p4/lpk+P9zokkbgXDPhJS/H3SBKAttNMxM9Yglgmgo5+qu1ngnoeGO2cOx54GXikoxdyzt3vnCtwzhXk5OR0c5i9Q0s4ws0L3uOJlSV886wJ3DpnktYXEIlDmWkp9E31U1bd5HUo+8WyaqgEaPsf/nCgtO0JzrmqNpsPAD+LYTy9TigcYeWHu3nl/XJeLCxja9Vevn/eJOadPs7r0ETkIMyM3H5Bymrip0QQy0SwHJhgZmNo7RV0JXBV2xPMLM85tyO6eRHQtSGAScI5x8K1Zfz4+ULKa5sI+I1ZYwfx3XMmcf7xeV6HJyKHEW+DymKWCJxzITO7CVhEa/fRh51zhWZ2B7DCOfcc8A0zuwgIAbuA62MVT29RVt3ID/+6jpfW7+S4YVn86MLJnD4xm8xgwOvQRKSThmSl89bmoxvtHAsxHUfgnFsILGy377Y2j28Fbo1lDL3Ja++X841H36M5HOHWOZO44dQxmi5CJAEN6ZfGztomwhEXnVrbWxpZnCAWb6rgy39cycQhGdwzdzqjs498ClwRiQ9D+qUTjjgq65rI7aapK7pC/04mgKVFlcz7wwrGD87gTzecpCQgkuDy4myBGiWCOPfOll3c8MgKRg3qw5++dJLmCxLpBdouWRkPlAjiWEVtE1/500ry+gf585dmMbCvkoBIb9B2UFlzKMLOmkaqG1o6/fyG5jCLCsv44bPrWF9a0+V41EYQp5xz/OCZtdQ1hXhs3qxPzJAoIolrYJ9UUv0+fvL3Ddz+fOusO+kBP7dflM/nCkZ0OBh0z95mXn2/nBcLd7J4UwUNLWGgdc6iBz5f0KV4lAji1LOrtvPi+p3cOmcSE3IzvQ5HRLqRz2f8xwXHUlxRz6C+qQzom8rCtTv43lNreeODSv770in0TU1hw44a3i6u4tX3y1m2ZRfhiCMnM43LZgzj3Ml5vLaxnN8v3UpFbVOX/llUIohDZdWN/OivhcwYNYAvnTbW63BEJAY+f/LoA7bnzhzJ7xZv5u6XNrFsyy6aWsLUNIYAGD84gy+fPpaz83OZOrz//nmRhvRL46E3t/Dse9v5t9OP/rNCiSDOOOe45ek1NIcj/OKKqXHRx1hEYs/vM7525nhmjR3Er17exLD+6cwaO4iTxg4kr196h88ZPziTaSP788TKbXzptDFHPb+YEkEccc7x3ws38M+NFfzownzGqJuoSNKZMWoAf7zhpE6ff8WMEXz/mbWsKalm6oj+R/We6jUUR+5+aRMPvLGF604exfWzR3sdjogkgAum5hEM+Hhi5bbDn3wQSgRx4revfMBvXy1i7swR/OjCyZpCWkQ6JSsY4NzJQ/jrqlIaoz2JjpQSgcciEcfdL23irpc2cem0YfzXv0zpsQUyRKR3uKJgBLWNIRYVlh3V89VG4KGaxha+/dgqXt5QzuUzhvPTS5UEROTInTx2EMP6p/OTv2/gz29/BEDfND93fe6ETg1EVYnAI5t21nLxPUv458YKbr8wnzsvP14ziYrIUfH5jO/NmcT4nAz8PmNvS4jXNlawYUfnRh2rROCB5lCEqx54GzD+8m+zmDlmoNchiUiCu2jqUC6aOhSA1dv2cPH8JZ1uM1Ai8MCbRRVU1jXz0HUFSgIi0u2CAT8AjS2RTp2vuggP/H1NGZnBFE6bkON1KCLSCwUDrR/tnS0RKBH0sOZQhBfXl/HZ/CGkpujHLyLdLz1aImhQIohPbxZVUNsY4vzjh3gdioj0Umn7q4aUCOLSvmqhU8erWkhEYmNf1VBTSG0EcUfVQiLSE1L9PsxUIohLqhYSkZ5gZgRT/EoE8UjVQiLSU9JT/fHRWGxm55rZRjMrMrNbDnHe5WbmzKxr663FMVULiUhPCqb4vB9HYGZ+YD4wB8gH5ppZfgfnZQLfAJbFKpZ4sKSoUtVCItJjgoH4qBqaCRQ554qdc83AAuDiDs77T+DnQGMMY/HcC+vKyExL4ZTx2V6HIiJJIC3g975EAAwD2q6UUBLdt5+ZTQNGOOf+dqgXMrN5ZrbCzFZUVFR0f6QxFgpHeGnDTs6cNJi0FL/X4YhIEggGfDSFvC8RdDSfstt/0MwH/BL4zuFeyDl3v3OuwDlXkJOTeA2tKz7cza76Zs49TtVCItIz0gN+Gpq9TwQlwIg228OB0jbbmcBxwD/NbCswC3iuNzYYv7CujNQUH5+amHhJTEQSUzDgpzEOSgTLgQlmNsbMUoErgef2HXTOVTvnsp1zo51zo4G3gYuccytiGFOPc87x0vqdnD4hm75pmuxVRHpGMBAHvYaccyHgJmARsAF43DlXaGZ3mNlFsXrfeLNuew3b9zRwzmRVC4lIzzmSAWUx/RfVObcQWNhu320HOfeMWMbilRcKd+D3GWcdm+t1KCKSRHqk15CZXXa0z00miwp3MnP0QAZ0Yt1QEZHukt5D4wh+2YXnJoWi8jqKyuvUW0hEelxrG0HsE0FH3UOljUWFZQB8drKqhUSkZwUDfkIRRyh8+OqhriQCd/hTklddU4g/v/0h00f2J69futfhiEiS2b9cZSfWJDhkY7GZraXjD3wDVN9xCHe9uJEdNY389qrpXociIkko2GaVsozDdF0/XK+hC7oppqSyatsefr90K9ecNIoZowZ4HY6IJKF9iaAzo4sPmQiccx8e7JiZLQFOOcLYer2WcIRbn17L4Mw0vnvuMV6HIyJJal8i6Mx8Q10ZRzCyC8/ttR56cwsbdtRw37UzyAoGvA5HRJJUMLruSWfGEnQlEaixuI29zSHuW1zMvYs3c87kXI0kFhFPtW0jOJzDNRZferBDgLrCAOGI46mVJfzixY2U1zZx3pQh/OfFx3kdlogkuY8TQddLBBce4tgh1xBIFo8s3codf1vPtJH9ufea6cwYNdDrkEREPu4+2tUSgXPuC90TUu/1QmEZx+Zl8fRXZ2OmMXYiEh/S9/Ua6kQiOOyAMjP7lJkdH338OTO7x8y+ZWZpXQ000dU0tvDuh7s585gcJQERiSvd2UYwHzgeCJrZRiADeAGYDTwMXN3FWBPa0qIqQhGnBWdEJO6kddfIYuBM51y+mQWB7cBg51zYzO4D1nQ10ES3eFMFGWkpTNegMRGJM/vHEXRD1VAjgHOuEfjQOReObjugpUtRJjjnHK9vqmD2uEEE/LFc6E1E5MgFU7qpaggYbGbfprW76L7HRLeTuj5kc0U92/c0cOOZ47wORUTkEwJ+w++zTjUWHy4RPEDrIvPtHwM8eHTh9Q6LN1UAcPqEpM6HIhKnzIxgSufWLT5c99Efd1tUvcziTRWMy+nLiIF9vA5FRKRDwU6uUna4XkMdri8c5Zxz/3mkgfUGjS1hlhVXcdVJmm5JROJXsJPrFh+uaqi+g319gRuAQUBSJoJlW3bRFIqo26iIxLW0gI/Grs4+6py7a99jM8sEbga+ACwA7jrY83q7xRsrSEvxMWvsIK9DERE5qPSAn8ZOrEfQmZHFA83sJ7SOG0gBpjvnvuecK+/Ec881s41mVmRmt3Rw/CtmttbMVpnZm2aWf9iIPdbQHOaFdTuYOWbg/n66IiLxKBjwd6pEcMhEYGZ3AsuBWmCKc+5259zuzgRgZn5gPjAHyAfmdvBB/xfn3BTn3AnAz4G7O/PaXvrFixsprW7kxjPGex2KiMghBQOd6zV0uBLBd4ChwH8ApWZWE/2qNbOawzx3JlDknCt2zjXTWp10cdsTnHNtX6Mvcb7GwcoPd/Hwki1cM2skJ49TtZCIxLdgSjf0GnLOdWXI7DBgW5vtEuCk9ieZ2deAbwOpwKc7eiEzmwfMAxg50pueOo0tYb775BqG9kvnljnHehKDiMiR6Gz30VjOjdDRdJyf+I/fOTffOTcO+B6tJY9PPsm5+51zBc65gpwcb3rq/PLlTRRX1PPTy6aQkdaVhd1ERHpGZ7uPxjIRlAAj2mwPB0oPcf4C4F9iGM9Re7u4igdeL+bKE0dwmkYSi0iCaG0j8LZEsByYYGZjzCwVuBJ4ru0JZjahzeb5wAcxjOeofFS1l6/+aSWjs/vy/fNVJSQiiaNbRhZ3hXMuZGY3AYsAP/Cwc67QzO4AVjjnngNuMrOzaJ3JdDdwXaziORq1jS186Q/LiTh46LoTyQoGvA5JRKTTggFft6xH0CXOuYXAwnb7bmvz+OZYvn9XhCOOmxesYnNFPX/44kzGZPf1OiQRkSMSTPETjjhawodOBppI/yD+97UiXn2/nNsvzOeU8dlehyMicsTSUzu3brESQQdqG1u4/41izpmcy7Unj/Y6HBGRo5LWyXWLlQg68Og7H1HbGOJrZ2r0sIgkrmBK60d802G6kCoRtNMUCvPQm1uYPW4Qxw/v73U4IiJHLagSwdH563ul7Kxp4iuf0hKUIpLYPk4EKhF0WiTi+N3rm8nPy+K0CWogFpHElh5QY/ERe3nDToor6vnyp8Zi1tEMGSIiiSMYaP2IV9XQEbjv9WKGD0jn/Cl5XociItJlaiM4QlV1Taz8cDdzZ44kxa8fi4gkvv0lgsOMLtYnXlRhaevSCNNGqKeQiPQOaSkqERyR9TtaE0H+0CyPIxER6R6qGjpChaU1DOufTv8+qV6HIiLSLfZNMaFE0EmFpdUqDYhIr7JvZLHGEXTC3uYQWyrrmaxEICK9SIrfR4rPVCLojA07anEO8vOUCESkd+nMcpVKBMD60moAJg/r53EkIiLdKxjwaWRxZxSW1tC/T4Ch/YJehyIi0q2CAT9NSgSHt35HDfl5WZpWQkR6nWDAT2NIieCQWsIR3i+rVUOxiPRKwYBPbQSHs7mijuZQhMlD1T4gIr1PMMWvXkOHsz46tYRKBCLSGwUDfjUWH05haQ1pKT7GZPf1OhQRkW6n7qOdUFhazaS8LM04KiK9UjDg87bXkJmda2YbzazIzG7p4Pi3zWy9ma0xs1fMbFQs42nPOcf60hpVC4lIr9VaIvAoEZiZH5gPzAHygblmlt/utPeAAufc8cCTwM9jFU9HSnY3UNMY0ohiEem1ggGfp+sRzASKnHPFzrlmYAFwcdsTnHOvOef2RjffBobHMJ5PKFRDsYj0cl73GhoGbGuzXRLddzA3AP/o6EcmfLwAAAsOSURBVICZzTOzFWa2oqKiotsCfGfLLtJSfByrEoGI9FLpqd72GupomK7r8ESza4AC4M6Ojjvn7nfOFTjnCnJycrotwKWbKykYPWD/4g0iIr1NMODHdfjJ+7FYJoISYESb7eFAafuTzOws4AfARc65phjGc4DKuibeL6tl9rjsnnpLEZEel5Zy+I/5WCaC5cAEMxtjZqnAlcBzbU8ws2nAfbQmgfIYxvIJSzdXAXDKeCUCEem9OlPjEbNE4JwLATcBi4ANwOPOuUIzu8PMLoqedieQATxhZqvM7LmDvFy3W1pUSWYwhSmaelpEerHOJIKUWAbgnFsILGy377Y2j8+K5fsfypLNlcwaOwi/TzOOikjvle5liSCefVS1l227Gjhl3CCvQxERialgwNs2gri1ZHMloPYBEen9PG0jiGdLiioZnJnG+MEZXociIhJTKhF0IBJxvLW5ilPGZ2tFMhHp9dJSVCL4hI07a6mqb2a22gdEJAmkpyoRfMKSIrUPiEjyUBtBB97aXMWY7L4M7Z/udSgiIjEX9HhkcVxaXVLN9JEDvA5DRKRHqETQTnltI5V1TZp2WkSShhJBO/sWqs9XIhCRJOH3GQH/oXtIJlci2NGaCLT+gIgkk8OVCpIqERSW1jB8QDr90gNehyIi0mOUCNrYUFqj9YlFJOnkZqUd8njSJIL6phBbquqZPFTTTotIcvnb10875PGkSQTvl9XinBqKRUTaS5pEsL60GlAiEBFpL3kSwY4a+qUHGNov6HUoIiJxJXkSQWkNk4dmacZREZF2kiIRhMIR3i+rVY8hEZEOJEUi2FJZT1MoovYBEZEOJEUiKNTUEiIiB5UUiWD9jhpSU3yMy9HSlCIi7cU0EZjZuWa20cyKzOyWDo6fbmbvmlnIzC6PVRzrS2s4JjeTgD8p8p6IyBGJ2SejmfmB+cAcIB+Ya2b57U77CLge+Eus4nDOsX6HppYQETmYlBi+9kygyDlXDGBmC4CLgfX7TnDObY0ei8QqiM0Vdeyqb+bYvMxYvYWISEKLZV3JMGBbm+2S6L4jZmbzzGyFma2oqKjo9POcc/zw2UIy01KYMyXvaN5aRKTXi2Ui6GjkljuaF3LO3e+cK3DOFeTk5HT6eY8t38ZbxVXcct4kcrM0olhEpCOxTAQlwIg228OB0hi+3wF21jTyXws3MHPMQOaeOLKn3lZEJOHEMhEsByaY2RgzSwWuBJ6L4fsd4La/rqMpFOGnl07B59O0EiIiBxOzROCcCwE3AYuADcDjzrlCM7vDzC4CMLMTzawEuAK4z8wKu+F9eWTpVhYV7uRbZ01krMYOiIgcUix7DeGcWwgsbLfvtjaPl9NaZdQtKmqbuPXptby8YSenjs/m304b010vLSLSa8U0EfSkVzbs5LtPrqGuKcR/nH8sXzxljKqEREQ6oVckgpZwhG88+h4jBvbhsXmzmJCrMQMiIp3VK+ZcWFNSTX1zmJs/M0FJQETkCPWKRPB2cRUAM8cM9DgSEZHE02sSwTG5mQzKSPM6FBGRhJPwiaAlHGHF1t3MGqvSgIjI0Uj4RLCmpJqGljCzxg7yOhQRkYSU8Ilg2Ra1D4iIdEXCJ4K3i3cxMTdD7QMiIkcpoRNBa/vALlULiYh0QUIngrXbq9nbrPYBEZGuSOhEoPEDIiJdl+CJoLV9IFvtAyIiRy1hE0FLOMJKtQ+IiHRZwk06t2lnLZ+565+EIo56tQ+IiHRZwiWCYMDPpCFZYDB73CDOOKbzaxiLiMgnJVwiGDmwD/Ovnu51GCIivUbCthGIiEj3UCIQEUlySgQiIklOiUBEJMkpEYiIJDklAhGRJKdEICKS5JQIRESSnDnnvI7hiJhZLbDR6zi6STZQ6XUQ3aC3XAfoWuJRb7kO8PZaRjnnOpyKIeFGFgMbnXMFXgfRHcxsRW+4lt5yHaBriUe95Togfq9FVUMiIklOiUBEJMklYiK43+sAulFvuZbech2ga4lHveU6IE6vJeEai0VEpHslYolARES6kRKBiEiSS6hEYGbnmtlGMysys1u8judImNlWM1trZqvMbEV030Aze8nMPoh+H+B1nB0xs4fNrNzM1rXZ12Hs1uo30Xu0xsziahWhg1zL7Wa2PXpvVpnZeW2O3Rq9lo1mdo43UX+SmY0ws9fMbIOZFZrZzdH9CXdfDnEtCXVfzCxoZu+Y2erodfw4un+MmS2L3pPHzCw1uj8tul0UPT7as+CdcwnxBfiBzcBYIBVYDeR7HdcRxL8VyG637+fALdHHtwA/8zrOg8R+OjAdWHe42IHzgH8ABswClnkdfyeu5Xbg3zs4Nz/6e5YGjIn+/vm9voZobHnA9OjjTGBTNN6Euy+HuJaEui/Rn21G9HEAWBb9WT8OXBnd/zvgq9HHNwK/iz6+EnjMq9gTqUQwEyhyzhU755qBBcDFHsfUVRcDj0QfPwL8i4exHJRz7nVgV7vdB4v9YuAPrtXbQH8zy+uZSA/vINdyMBcDC5xzTc65LUARrb+HnnPO7XDOvRt9XAtsAIaRgPflENdyMHF5X6I/27roZiD65YBPA09G97e/J/vu1ZPAZ8zMeijcAyRSIhgGbGuzXcKhf1nijQNeNLOVZjYvui/XObcDWv8YgMGeRXfkDhZ7ot6nm6JVJg+3qaJLiGuJVilMo/U/0IS+L+2uBRLsvpiZ38xWAeXAS7SWVvY450LRU9rGuv86osergUE9G3GrREoEHWXKROr7eopzbjowB/iamZ3udUAxkoj36V5gHHACsAO4K7o/7q/FzDKAp4BvOudqDnVqB/vi/VoS7r4458LOuROA4bSWUo7t6LTo97i5jkRKBCXAiDbbw4FSj2I5Ys650uj3cuAZWn9Jdu4rnke/l3sX4RE7WOwJd5+cczujf8AR4AE+rmaI62sxswCtH5x/ds49Hd2dkPelo2tJ1PsC4JzbA/yT1jaC/ma2b163trHuv47o8X50vtqyWyVSIlgOTIi2wKfS2rjynMcxdYqZ9TWzzH2Pgc8C62iN/7roadcBf/UmwqNysNifAz4f7aUyC6jeV1URr9rVlV9C672B1mu5Mtq7YwwwAXinp+PrSLQu+SFgg3Pu7jaHEu6+HOxaEu2+mFmOmfWPPk4HzqK1veM14PLoae3vyb57dTnwqou2HPc4r1vaj+SL1p4Pm2itd/uB1/EcQdxjae3lsBoo3Bc7rfWBrwAfRL8P9DrWg8T/KK1F8xZa/4u54WCx01rcnR+9R2uBAq/j78S1/DEa6xpa/zjz2pz/g+i1bATmeB1/m7hOpbUaYQ2wKvp1XiLel0NcS0LdF+B44L1ovOuA26L7x9KaqIqAJ4C06P5gdLsoenysV7FrigkRkSSXSFVDIiISA0oEIiJJTolARCTJKRGIiCQ5JQIRkSSXiIvXi/QIM9vXFRNgCBAGKqLbe51zsz0JTKSbqfuoSCeY2e1AnXPuF17HItLdVDUkchTMrC76/QwzW2xmj5vZJjP7qZldHZ2Xfq2ZjYuel2NmT5nZ8ujXKd5egcjHlAhEum4qcDMwBbgWmOicmwk8CHw9es6vgV86504ELoseE4kLaiMQ6brlLjpvj5ltBl6M7l8LnBl9fBaQ32a6+Swzy3St8++LeEqJQKTrmto8jrTZjvDx35gPONk519CTgYl0hqqGRHrGi8BN+zbM7AQPYxE5gBKBSM/4BlAQXW1rPfAVrwMS2UfdR0VEkpxKBCIiSU6JQEQkySkRiIgkOSUCEZEkp0QgIpLklAhERJKcEoGISJL7/8w6Om85JSk9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ev.nbll(time_grid).plot()\n",
    "plt.ylabel('NBLL')\n",
    "_ = plt.xlabel('Time')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Integrated scores\n",
    "\n",
    "The two time-dependent scores above can be integrated over time to produce a single score [Graf et al. 1999](https://onlinelibrary.wiley.com/doi/abs/10.1002/%28SICI%291097-0258%2819990915/30%2918%3A17/18%3C2529%3A%3AAID-SIM274%3E3.0.CO%3B2-5). In practice this is done by numerical integration over a defined `time_grid`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.17627978358784843"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ev.integrated_brier_score(time_grid) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5240259319069138"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ev.integrated_nbll(time_grid) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
